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Measuring  Leader  Attributes  in  the  Army  Reconnaissance  Course 


Introduction 

In  201 1,  the  Army  introduced  the  Army  Learning  Model  (ALM)  in  the  U.S.  Anny 
Learning  Concept  (ALC)  for  2015  (U.S.  Department  of  the  Anny,  2011)  and  put  forth  ideas  for  a 
more  Soldier-centered  approach  to  learning.  At  the  center  of  the  ALM  is  the  concept  of 
adaptability  in  that  (a)  the  Anny  must  develop  adaptive  Soldiers;  (b)  training  content  must  be 
adaptive  such  that  training  occurs  outside  of  the  classroom  and  adjusts  to  operational  needs;  and 
(c)  content,  learning  methods,  and  technologies  continuously  adapt  based  on  Soldier 
perfonnance.  Among  other  changes,  to  enable  those  different  forms  of  adaptation,  the  Soldier- 
centered  environment  requires  a  shift  from  instructor-based  classroom  methods  of  teaching  to  a 
blended,  more  individualized  model  that  incorporates  the  three  pillars  of  leader  development: 
institutional  instruction,  self-development,  and  operational  experience  (U.S.  Department  of  the 
Anny,  2013).  In  such  an  environment,  learning  experiences  are  tailored  to  individual  needs, 
creating  more  engaging  and  effective  training.  In  addition,  training  must  be  focused  on  the 
development  of  21st  Century  Soldier  Competencies  (U.S.  Department  of  the  Army,  2011),  which 
are  core  competencies  that  provide  the  foundation  for  the  development  of  operational 
adaptability.  The  ALC  2015  outlines  nine  21st  Century  Soldier  Competencies,  including 
character  and  accountability;  adaptability  and  initiative;  and  critical  thinking  and  problem 
solving.  The  focus  on  such  learning  outcomes  formalizes  existing  objectives  in  some 
institutional  training  courses  and  incorporates  new  goals  in  other  courses  which  were  previously 
focused  primarily  on  tactical  and  technical  competence. 

Assessment  is  a  specific  challenge  related  to  the  execution  of  the  ALM.  “The  importance 
of  incorporating  valid  and  reliable  assessments  in  the  2015  learning  model  cannot  be  overstated” 
(U.S.  Department  of  the  Army,  201 1,  p.  21).  Without  appropriate,  effective  assessment 
methodologies,  instructors  will  not  possess  the  data  necessary  to  provide  the  student  with 
learning  experiences  that  speak  to  his/her  needs.  The  current  work  focused  on  the  development 
of  tools  to  help  instructors  assess  the  development  of  21st  Century  Soldier  Competencies 
effectively  and  reliably  across  students  and  instructors  over  time.  Assessing  the  development  of 
competencies  such  as  adaptability  and  initiative  is  certainly  not  as  straight  forward  as  the 
assessment  of  a  skill  like  marksmanship  where  there  are  well-defined  performance  standards. 

The  tools  described  in  this  report  provide  observer-based  methods  for  assessing  specific  21st 
Century  Soldier  Competencies. 

Current  Research 

One  example  of  a  course  that  has  implemented  ALM  principles  into  its  training  is  the 
Anny  Reconnaissance  Course  (ARC).  The  ARC  aims  to  develop  confident  and  agile  Soldiers 
capable  of  operating  under  unpredictable  combat  and  training  situations.  Specifically,  along  with 
technical  and  tactical  reconnaissance  skills,  the  ARC  aims  to  develop  Soldier  competencies 
similar  to  those  described  in  the  ALM  (c.fi,  U.S.  Department  of  the  Anny,  2011;  Table  1).  The 
ARC  instructors  apply  the  principles  put  forth  in  the  ALM  to  create  learning  experiences  that 
allow  students  to  develop  these  leader  attributes. 
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Table  1 


Leader  Attributes  Targeted  in  the  ARC 


Leader  Attribute 

Definition 

Accountability 

Takes  responsibility  for  own  and  team’s  actions  and  consequences 

Adaptability 

Manages  changing  requirements  for  balancing  unit  recon, 
surveillance,  and  security  with  mission  accomplishment 

Anticipation 

Foresees  future  requirements  and  conditions 

Confidence 

Believes  in  own  and  team’s  ability  to  handle  tactical  situations 

Initiative 

Thinks  and  acts  without  being  urged 

Problem  Solving 

Solves  problems  by  applying  deliberate  thought 

Risk  Management 

Assesses  the  situation  against  the  mission  and  makes  a  decision  - 
effectively  balances  mission  requirements  and  risk 

There  are  several  issues  that  make  assessment  challenging  within  the  ARC.  First,  as 
previously  noted,  it  is  difficult  to  accurately  and  consistently  assess  the  development  of  leader 
attributes  in  students.  Although  the  leader  attributes  are  well-defined  within  the  ARC 
curriculum,  there  is  still  a  degree  of  subjectivity  associated  with  the  assessment  of  a  competency 
such  as  confidence.  Second,  instructors  who  are  observing  and  rating  student  performance  rotate 
out  during  course  exercises  creating  a  need  to  communicate  the  levels  at  which  students  are 
perfonning.  For  example,  when  instructor  A  is  replaced  by  instructor  B  during  an  exercise, 
instructor  B  needs  to  understand  instructor  A’s  observations  and  assessments  of  student 
perfonnance  during  instructor  A’s  shift.  Finally,  within  the  ARC,  the  main  mechanism  for 
tracking  student  progress  is  through  paper-based  assessments;  a  method  is  needed  to  increase  the 
reliability  and  sustainability  of  those  assessments  and  relieve  some  of  the  burden  created  through 
a  paper-based  approach.  Tracking  progress  on  the  leader  attributes  and  other  relevant 
perfonnance  outcomes  is  important  so  that  course  events  can  be  adjusted  to  continually 
challenge  and  develop  the  students  based  on  prior  perfonnance. 

To  aid  the  ARC  instructors  with  those  assessment  challenges,  this  research  focused  on  the 
development  of  two  products:  (a)  objective  behaviorally-based  measures  of  the  leader  attributes 
within  the  context  of  reconnaissance  missions  to  promote  consistency  between  and  across 
students  and  instructors;  and  (b)  a  technology-supported  perfonnance  assessment  toolkit  to  allow 
instructors  to  more  easily  track  critical  performance  information  throughout  the  course.  The 
remainder  of  this  document  describes  the  development  and  evaluation  of  these  products. 
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Measure  Development 


Initial  Measure  Development 

To  aid  the  instructors  in  the  assessment  of  leader  attributes,  we  leveraged  Aptima’s 
COmpetency-based  Measures  for  Perfonnance  Assessment  Systems  (COMPASSsm) 
methodology,  which  is  an  approach  to  developing  valid,  reliable  and  sensitive  measures  of  team 
and  individual  perfonnance  (see  MacMillan,  Entin,  Morley,  &  Bennett,  2013).  COMPASS 
employs  an  iterative  series  of  three  workshops  with  Army  personnel  as  the  subject  matter  experts 
(SMEs)  to  develop  and  initially  validate  perfonnance  measures.  The  purpose  of  the  workshops 
is  to  obtain  specific  behavioral  indicators  in  relation  to  perfonnance  objectives  (in  this  case,  the 
leader  attributes).  The  COMPASS  process  is  consistent  with  the  development  of  Behavioral 
Anchored  Rating  Scales  (BARS;  Smith  &  Kendall,  1963),  or  perhaps  even  more  similar, 
Behavioral  Summary  Scales  (BSS;  Borman,  Hough,  &  Dunnette,  1976).  Both  BARS  and  BSS 
are  “anchored  with  behavioral  descriptions  of  effective  and  ineffective  perfonnance  to  guide 
raters’  evaluations”  (Pulakos,  1997,  p.  300).  BSS  tend  to  be  more  generalizable  than  BARS 
because  they  provide  multiple  behavioral  descriptors  at  different  perfonnance  levels.  Such 
rating  scales,  which  focus  on  observable  behaviors,  are  useful  for  understanding  performance 
challenges  (Beaubien,  Goodwin,  Milanovich,  Baker,  &  Smith-Jentsch,  2004). 

Senior  ARC  instructors  served  as  SMEs  for  the  COMPASS  workshops.  ARC  instructors 
typically  hold  the  rank  of  Staff  Sergeants  (SSGs),  with  approximately  10-12  years  in  service. 
Most  instructors  have  been  in  leadership  positions  as  scouts  or  senior  scouts  in  reconnaissance 
units.  ARC  instructors  are  often  graduates  of  the  Advanced  Leaders  Course  (ALC)  and  the  Army 
Basic  Instructor  Course  (ABIC).  More  senior  instructors  hold  the  rank  of  Sergeant  First  Class 
(SFC)  and  possess  12-15  years  of  experience.  They  may  have  held  platoon  sergeant  positions 
and  likely  are  graduates  of  the  Senior  Leaders  Course  (SLC)  in  addition  to  ABIC.  Both  SSGs 
and  SFCs  participated  in  the  workshops. 

During  COMPASS  Workshop  1,  seven  instructors  provided  information  about  the 
behaviors  they  look  for  to  assess  each  leader  attribute.  To  elicit  that  information,  the  instructors 
were  asked  to  verbally  describe  what  behaviors  they  look  for  in  assessing  each  leader  attribute 
within  the  context  of  three  training  events  -  Planning,  Bridge  Reconnaissance,  and  Patrolling. 
Next,  performance  levels  or  anchors  of  the  observed  behaviors  were  specified  (i.e.,  excellent, 
average,  and  poor  performance).  The  workshop  resulted  in  a  list  of  behavioral  indicators  for 
each  attribute  categorized  from  novice  to  expert  and  an  ordering  of  the  behaviors  within  each 
category  based  on  the  difficulty  of  the  associated  tasks.  Following  the  workshop,  behavioral 
indicators  for  each  leader  attribute  for  the  three  training  events  were  synthesized  by  extracting 
common  indicators  within  each  attribute  and  across  events.  Those  indicators  were  then 
developed  into  behavioral  statements  that  would  allow  instructors  to  simply  indicate  whether  or 
not  they  observed  a  student  performing  that  specific  behavior.  For  example,  for  the  attribute  of 
Confidence,  instructors  indicated  that  poor  performance  was  characterized  by  students  not 
knowing  where  to  start  solving  a  problem  (e.g.,  not  taking  action)  and  continuously  seeking 
instructor  and  peer  approval.  The  infonnation  received  about  the  indicators  of  Poor  Confidence 
led  to  the  development  of  six  behavioral  statements: 
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•  Continuously  seeks  input  from  peers  without  deciding/acting 

•  Relies  on  only  the  "strong"  students  for  support 

•  Changes  answers  when  asked  "why,"  chooses  not  to  defend  logic 

•  Loses  momentum  or  takes  too  much  time  to  decide 

•  Overly  confident;  manages  all  aspects  of  planning  and  execution  without  accounting  for 
subordinate  suggestions 

•  Subordinates  overly  question  decisions 

Similar  statements  were  developed  to  describe  Poor,  Average,  and  Excellent  performance  on 
each  of  the  attributes. 

The  goals  of  Workshop  2  were  to  ensure  the  completeness  of  the  statements,  reduce 
redundancy,  and  obtain  input  on  the  order  of  the  behavioral  statements  to  inform  how  they 
should  be  grouped.  The  workshop  resulted  in  wording  changes  to  some  of  the  behavioral 
statements  and  deletions  of  redundant  statements.  The  revised  behavioral  statements  were  used 
to  develop  the  performance  measures  for  each  leader  attribute.  The  scale  on  each  measure  was 
revised  to  fit  the  assessment  scale  currently  used  in  the  ARC  (i.e.,  -2  to  +2)  as  opposed  to  poor, 
average,  and  excellent  performance. 

The  focus  of  Workshop  3  was  to  review  the  leader  attribute  measures  for  completeness 
and  clarity.  Over  the  course  of  about  three  hours,  five  instructors  reviewed  the  measures  and 
revised  for  clarity.  As  an  example,  the  full  Confidence  measure  is  shown  in  Figure  1.  The 
measures  were  then  tested  during  an  evaluation,  which  is  described  in  the  next  section. 


CONFIDENCE: 


Believes  In  Own  And  Team’s  Ability  To  Handle  Tactical  Situations 


Continuously 
seeks  input  from 
peers  without 
deciding/  acting 

Fails  to  act  when 
challenged 


Runs  platoon  as  a 
democracy 

Changes  answers 
when  asked  “why," 
chooses  not  to 
defend  logic 

Overly  confident; 
manages  all 
aspects  of 
planning  and 
execution  without 
accounting  for 
subordinate 
suggestions 


Incorporates  input 
from  subordinate 
leadership  with 
instructor  approval 

Task,  purpose, 
intent  is  over¬ 
complicated  or  too 
vague,  suggesting 
uncertainty 

Loses  momentum 
or  takes  too  much 
time  to  decide 


Considers 
recommendations 
from  subordinate 
leadership  and 
incorporates  as 
appropriate 

Provides  evidence 
when  asked  “why” 

Utilizes  the 
“strong"  students 
for  support  when 
necessary 

Effectively  conveys 
task,  purpose, 
intent  under 
moderate  levels  of 
stress 


Maintains 
command 
and  control 
under  difficult 
conditions  (e.g., 
extreme  stress, 
uncertainty)  and 
effectively  conveys 
task,  purpose, 
intent 


Figure  1.  Example  leader  attribute  measure. 
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Measure  Evaluation 


Measures  were  evaluated  on  their  (a)  usability  and  utility  to  the  ARC  instructors  and  (b) 
ability  to  reliably  reflect  leader  attributes.  Prior  to  using  the  measures  during  the  ARC’S 
culminating  exercise,  the  instructors  assigned  to  be  involved  in  the  evaluation  participated  in  a 
planning  meeting  with  the  research  team.  The  anchors  were  reviewed  during  this  meeting  to 
ensure  that  all  raters  were  using  the  measures  with  the  same  frame  of  reference. 

To  evaluate  the  measures,  approximately  12  hours  of  observations  were  conducted  over 
the  course  of  two  days.  During  the  observations,  the  research  team’s  two  SMEs  (retired  Anny 
noncommissioned  officers)  were  paired  with  an  instructor  to  rate  the  students  assigned  to  the 
Platoon  Leader  and  Platoon  Sergeant  positions  using  the  leader  attribute  measures;  their  ratings 
were  later  compared  for  level  of  agreement.  Following  the  observations  and  evaluation  period, 
the  instructors  completed  a  reaction  questionnaire  to  assess  the  usability  and  utility  of  the 
measures  (see  Appendix  A  for  the  measure).  Eight  questions  were  asked  (e.g.,  I  think  that  I 
would  like  to  use  these  measures  and  anchors  frequently,  I  thought  that  there  was  too  much 
inconsistency  in  the  anchors  [reverse  coded])  to  which  the  instructors  responded  on  a  5-point 
scale  ( 1  =  Strongly  Disagree;  5  =  Strongly  Agree).  The  evaluations  occurred  during  key  events 
during  the  training  (planning,  infiltration,  and  bridge  reconnaissance).  The  instructors  and  the 
team’s  SMEs  each  made  their  ratings  independently  of  one  another,  and  as  possible,  the  rating 
forms  were  collected  directly  following  a  key  event.  However,  due  to  instructor  schedules, 
several  of  the  rating  forms  were  not  collected  immediately. 

Rater  agreement  results.  To  conduct  the  reliability  assessment  of  the  measures,  rater 
agreement  data  were  calculated  by  comparing  paired  ratings  between  the  ARC  instructors  and 
the  research  team  SMEs.  For  each  of  the  seven  attributes  (Accountability,  Adaptability, 
Anticipation,  Confidence,  Initiative,  Problem  Solving,  and  Risk  Mitigation)  between  two  and  six 
pairs  of  ratings  were  available.  The  variability  in  the  number  of  paired  ratings  for  each  attribute 
was  due  to  missing  data  as  well  as  the  students  not  displaying  some  attributes  during  the  training 
events.  In  total,  33  pairs  of  ratings  were  available  to  assess  interrater  agreement.  Across  all  33 
pairs,  percent  agreement,  as  calculated  by  the  paired  ratings  being  within  one  point  of  each  other, 
was  72.73%.  However,  when  examined  by  attribute,  percent  agreement  (again  defined  as  the 
paired  ratings  being  within  one  point  of  each  other)  ranged  from  33%  to  100%.  Table  2  displays 
the  agreement  on  each  of  the  attributes,  as  well  as  the  number  of  paired  ratings  available  for  each 
attribute.  As  can  be  seen,  the  two  attributes  with  the  lowest  agreement  are  Adaptability  and 
Confidence.  In  general,  the  percent  agreement  is  good  (with  the  exception  of  the  Adaptability 
ratings)  and  demonstrates  that  the  measures  generally  promote  consistent  assessment  of  the  ARC 
leader  attributes. 
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Table  2 


Percent  Agreement  for  Each  Leader  Attribute 


Attribute 

Number  of  Paired 
Ratings 

Percent  Agreement  (within  1  point  or 
better) 

Initiative 

4 

100% 

Risk  Mitigation 

2 

100% 

Accountability 

6 

83% 

Anticipation 

5 

80% 

Problem  Solving 

5 

80% 

Confidence 

5 

60% 

Adaptability 

6 

33% 

Following  the  evaluation,  one  final  workshop  was  held  with  the  instructors  to  revise  the  e 
measures  based  on  the  evaluation  data  (e.g.,  reach  100%  agreement  for  the  scales  with  low  levels 
of  interrater  agreement).  In  discussions  between  the  research  team’s  SMEs  and  the  instructors 
who  made  the  ratings,  it  became  apparent  that  although  many  of  the  same  student  behaviors  were 
noted  by  both  individuals  in  each  pair  of  observers  close  overlap  in  how  each  attribute  is  defined 
may  have  contributed  to  the  low  levels  of  agreement.  For  example,  while  the  instructor  and  the 
research  team’s  SME  noticed  the  same  behavior  within  students,  the  instructor  used  that  behavior 
as  an  indication  of  Anticipation,  while  the  research  team  SME  coded  that  same  behavior  as  an 
indication  of  Adaptability.  In  this  case,  the  overlap  is  likely  due  to  Anticipation  being  a  key 
element  of  adaptive  behavior,  and  because  of  that,  anchors  tended  to  reflect  similar  behaviors. 
This  overlap  across  attributes  speaks  to  the  need  for  a  tool  to  help  instructors  consistently  rate 
students.  In  addition,  the  lack  of  agreement  on  some  of  the  attributes  indicates  that  for  the 
measures  to  be  maximally  effective,  the  instructors  should  become  more  familiar  with  the 
behaviors  listed  for  each  attribute.  It  should  be  noted  that  prior  to  the  evaluation,  no  practice 
ratings  were  completed.  A  best  practice  for  implementing  these  measures  in  the  course  may  be 
to  include  such  training  to  ensure  high  reliability.  Although  slight  revisions  were  made  to  two  of 
the  anchors  (Adaptability  and  Confidence)  following  this  final  workshop,  no  major  changes  were 
made. 


Usability  and  utility  results.  For  the  usability  and  utility  evaluation  of  the  measures,  the 
average  rating  was  4.28  out  of  5  ( SD  =  .64),  indicating  that  the  anchors  were  perceived 
favorably.  In  addition,  qualitative  comments  validated  the  quantitative  ratings  and  substantiated 
the  expectation  that  the  rating  form  increases  the  consistency  in  how  the  instructors  rated 
students.  In  particular,  one  instructor  noted  that  he  “found  the  anchors  to  be  very  effective  at 
ensuring  we,  as  instructors,  rate  attributes  on  the  same  guidelines.”  Item-level  descriptive 
statistics  are  found  in  Appendix  B. 

Final  Measures 

The  final  measures  are  in  Appendix  C.  As  discussed,  each  of  the  seven  leader  attributes 
have  behavioral  descriptions  that  range  from  a  -2  (to  indicate  that  the  student  is  engaging  in 
behaviors  that  do  not  meet  the  intent  of  that  leader  attribute)  to  +2  (to  describe  the  behaviors  that 
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leaders  should  be  seeing  when  a  student  is  excelling  at  that  attribute).  A  zero  rating  indicates 
that  the  student  is  performing  at  baseline  per  Army  doctrine.  The  goal  of  the  measures  is  to 
provide  guidelines  for  instructors  to  orient  them  toward  important  behaviors.  The  measures  may 
be  especially  useful  for  more  novice  instructors  who  do  not  have  as  much  expertise  in  assessing 
the  leader  attributes  as  more  experienced  instructors.  When  student  assessments  are  made  based 
on  the  behaviors  articulated  in  developed  anchors,  instructors  should  be  rating  students  more 
consistently. 


Tool  Development 

In  addition  to  the  performance  measures,  the  ARC  Performance  Assessment  Toolkit 
(ARC-PAT)  was  developed  to  support  the  ARC  instructors  and  house  the  perfonnance  measures. 
The  purpose  of  the  toolkit  is  to  provide  mobile,  digital  data  capture  solutions  with  easy  to  use 
interfaces  that  reduce  redundancies  and  overall  workload.  This  goal  was  accomplished  through 
the  use  of  existing  observer-based  measurement  tools  (Aptima’s  SPOTLITE  tool)  and  database 
software  customized  to  the  ARC’S  use  case  and  assessment  measures. 

Requirements  Elicitation  Workshops 

To  develop  a  useful  and  usable  tool  that  met  the  needs  of  the  ARC  instructors, 
requirements  were  gathered  during  a  series  of  four  workshops.  Four  to  six  instructors 
participated  in  each  workshop,  with  several  instructors  participating  in  multiple  workshops.  The 
primary  goal  of  the  workshops  was  to  develop  a  set  of  tools  that  would  support  the  ARC 
instructors  in  their  assessments  and  yield  a  reduction  in  overall  workload. 

The  primary  goal  of  the  first  workshop  was  to  develop  an  understanding  of  the 
instructors’  responsibilities  in  terms  of  assessment  of  students.  The  discussion  centered  on 
identifying  what  instructors  must  do  on  a  day-to-day  basis  to  document  and  report  on  student 
progression.  This  discussion  yielded  a  set  of  requirements  from  which  a  series  of  static  mockups 
were  produced. 

The  second  and  third  workshops  focused  on  review  and  revision  of  the  mockups 
generated  after  the  first  workshop.  The  second  workshop  featured  an  initial  set  of  mockups 
representing  the  primary  interface  of  the  mobile  field  tool  and  key  interface  elements.  These 
elements  were  presented  and  described  to  instructors.  The  mockups  were  presented  in  order  of 
expected  workflow  to  allow  the  instructors  to  conceptualize  use  of  the  tool  to  collect  data  and 
complete  assessments  for  their  students.  During  the  presentation,  instructors  were  encouraged  to 
ask  questions  and  provide  comments  with  respect  to  the  mockups  and  the  tool  concepts. 

The  third  workshop’s  goal  was  to  finalize  interface  and  functional  design  ideas  through  a 
revised  set  of  mockups.  The  revisions  reflected  feedback  received  during  the  second  workshop. 
Again,  the  mockups  were  presented  and  described  in  a  fashion  consistent  with  the  anticipated 
workflow,  and  participant  instructors  were  encouraged  to  comment  or  question  the  approach. 
Revised  mockups  are  in  Appendix  E. 

The  fourth  workshop  had  two  goals.  The  first  was  to  demonstrate  a  working  prototype  of 
the  ARC  Field  Tool  (FT)  and  verify  that  its  function  and  features  aligned  with  expectations  of 
the  instructors.  The  second  goal  was  to  introduce  mockups  and  storyboards  of  the  Integrated 


7 


Database.  Until  this  workshop,  the  database  had  only  been  discussed  with  respect  to  the 
requirements  identified  in  the  previous  workshops.  During  this  workshop,  instructors  were 
invited  to  try  out  the  FT  and  provide  comment.  The  database  was  presented  in  the  same  fashion 
as  mockups  for  the  FT  in  prior  workshops,  and  instructors  were  encouraged  to  question, 
comment  and  otherwise  guide  revisions  or  changes  to  the  tool  and  database. 

Tool  Functionality  Development 

After  all  requirements  had  been  derived  and  the  instructors  were  pleased  with  the  mock- 
ups,  development  of  the  tool  commenced.  Tool  development  followed  an  iterative  process 
broken  up  into  three- week  sprints.  Each  sprint  contributed  to  the  evolution  of  the  software  and 
progression  towards  a  final  working  prototype.  The  first  sprint  focused  on  development  of  base 
graphical  user  interface  (GUI)  architecture  and  primary  user  controls.  The  second  and  third 
sprints  added  additional  controls  and  features  and  brought  the  tool  to  test-ready  state.  The  fourth 
sprint  followed  initial  user  demonstrations  where  feedback  was  elicited  and  refinements  and 
revisions  to  the  tool  were  implemented. 

Key  tool  components.  The  ARC-PAT  is  an  integrated  mobile  data  collection  and 
perfonnance  assessment  system.  The  system  is  designed  for  ARC  instructors  to  capture  and  rate 
student  perfonnance  as  they  progress  through  the  27-day  course.  Data  can  be  collected  on 
student  behaviors  reflecting  the  expected  course  outcomes  and  the  leader  attributes  (as  described 
above  and  in  Appendix  B)  in  both  classroom  and  field  events.  The  ARC -FT  is  a  digitized 
version  of  the  notebook  carried  by  instructors  and  the  paper  assessment  form  required  for 
documentation  of  student  progression.  The  ARC-FT  enables  ARC  instructors  to  capture  critical 
performance  metrics  for  students  that  reflect  learning  and  progression  within  the  ARC.  It  also 
supports  multi-media  capture  of  student  behavior  including  photo,  video  and  voice-text  and 
allows  users  to  attach  those  data  to  assessments  of  the  course  outcomes  and  leader  attributes.  A 
simple  trending  interface  within  the  app  provides  users  with  data  on  students  over  the  duration  of 
the  course.  The  ARC-FT  app  is  integrated  with  the  ARC  Database  through  a  WiFi  connection. 

The  Integrated  Database  is  a  web-based  software  tool  that  stores  and  manages  student 
perfonnance  data.  It  features  a  user  interface  that  supports  the  display  and  review  of 
perfonnance  data  across  students,  units,  classes,  and  even  years.  The  database  interface  offers 
tools  for  reviewing  the  data  to  identify  trends  and  patterns  that  will  help  instructors  and 
leadership  manage  the  course.  It  also  fonnats  student  performance  data  to  minor  the  assessment 
forms  used  by  the  instructors,  which  can  be  printed  and  stored  according  to  cunent  best 
practices.  It  is  hosted  on  a  desktop  and  offers  a  wireless  local  area  network  (LAN).  The  LAN 
allows  configurations  (e.g.,  student  rosters)  to  be  loaded  onto  the  field  tool  and  perfonnance 
assessment  data  to  be  uploaded,  stored,  and  printed.  A  diagram  of  the  ARC-PAT  architecture  is 
shown  in  Figure  2. 
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Figure  2.  Architecture  diagram  of  ARC-PAT. 


Tool  Evaluation 

The  team  conducted  two  evaluations  of  the  tool.  The  first  was  a  supported  field  test  and 
the  second  was  an  unsupported  leave-behind.  Both  evaluation  opportunities  occurred  during 
regularly  scheduled  course  assessment  events.  The  primary  goal  of  both  evaluations  was  to 
informally  evaluate,  with  SME  and  perspective  user  input,  the  ability  of  the  ARC -PAT  to  support 
data  collection  and  measurement  during  trainee  assessment.  In  order  for  an  assessment  system  to 
be  efficient  and  effective,  an  instructor/unit  leader  must  be  able  to  use  the  system  in  an  intuitive 
and  easy  manner  while  observing  the  Soldier  behaviors.  For  research  purposes,  usability  was 
defined  as  a  measure  of  the  effectiveness,  efficiency,  and  satisfaction  with  which  specified  users 
can  achieve  goals  in  a  particular  environment.  The  tenn  effectiveness  refers  to  the  utility  of 
ARC-PAT  in  collecting,  summarizing,  measuring,  and  reviewing  behaviors  during  training. 
Efficiency  and  satisfaction  were  operationalized  as  the  users’  ability  to  perform  these  tasks 
quickly  and  relatively  error  free.  A  secondary  goal  of  the  evaluation  was  to  identify 
opportunities  for  enhancement  of  the  tools  based  on  usability  issues  or  shortfalls  in  the  toolkit’s 
functionality. 

Seven  ARC  instructors  participated  in  the  first  field  test,  and  five  participated  in  the 
unsupported  leave-behind,  with  some  overlap  in  instructors  across  the  two  events.  During  both 
events,  instructors  used  the  tools  to  evaluate  student  perfonnance  during  graded  course  events. 
Prior  to  the  start  of  the  evaluations,  instructors  were  given  access  to  the  tool,  and  training 
occurred  on  both  the  functionality  of  the  tool  and  the  measures.  During  the  training,  a  written 
user  guide  (see  Appendix  F)  was  provided  that  pointed  out  the  intended  use  of  the  toolkit  and 
identified  the  main  operations  and  features  that  facilitate  its  use.  Following  the  training,  the 
research  team  encouraged  the  instructors  to  familiarize  themselves  with  the  functions  of  the  tool 
prior  to  using  the  tool  during  the  field  training  exercises  (FTXs)  and  other  graded  course  events. 

The  first  evaluation  was  a  brief,  fully  supported  test  of  the  ARC-FT  during  one  of  the 
ARC’s  FTXs.  This  fully  supported  test  consisted  of  the  research  team  being  embedded  with  the 
ARC  instructors.  Although  researchers  provided  technical  and  user  support  during  the  event, 
support  was  restricted  to  answering  instructor  questions  and  providing  assistance  as  requested, 
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rather  than  demonstrating  the  tool’s  potential  capabilities.  This  strategy  allowed  the  instructors 
to  identify  problem  areas  that  needed  to  be  addressed. 

The  second  evaluation  consisted  of  a  minimally  supported  leave-behind  with  the 
complete  ARC -PAT  available  for  instructor  use.  ARC-PAT  was  left  with  the  instructors  for  one 
full  class  cycle,  or  about  40  days.  During  this  time,  no  researchers  were  present;  however, 
routine  check-ins  were  performed  to  ensure  that  the  software  was  operating  as  intended  and  to 
answer  instructors’  questions.  Instructors  could  use  the  system  as  much  or  as  little  as  they 
desired  with  no  encouragement  or  additional  instruction  offered  by  the  research  team.  The  goal 
of  this  evaluation  was  to  identify  any  other  outstanding  problems  that  needed  to  be  addressed 
prior  to  the  end  of  the  research  project.  Additionally,  the  realistic  conditions  of  the  leave-behind 
pennitted  a  more  rigorous  test  of  the  tool’s  capabilities,  functions,  and  interactivity  with  the 
instructors. 

Shortly  after  both  evaluations,  instructors  were  asked  to  complete  several  assessment 
instruments:  a  usability  questionnaire,  a  set  of  usability  and  utility  statements,  and  a  five-item 
structured  interview  aimed  at  identifying  the  extent  to  which  the  tools  sufficiently  met  a  set  of 
key  design  principles,  and  thus  achieved  the  goals  of  effectiveness,  efficiency,  and  satisfaction. 

The  usability  questionnaire  is  a  standard  set  of  10  general  statements  about  the  tool  to 
which  respondents  can  agree  or  disagree  on  a  4-point  scale  (Strongly  Disagree,  Disagree,  Agree, 
Strongly  Agree).  The  results  of  this  questionnaire  point  to  the  general  approval  or  disapproval  of 
the  tool  but  do  not  point  to  any  specific  elements  of  the  tool  that  may  be  desirable  or  undesirable. 
More  specific  queries  are  featured  in  the  usability  statements.  Separate  usability  questionnaires 
were  developed  for  the  ARC -FT  and  Integrated  Database.  Both  questionnaires  (Appendix  G)  are 
adapted  from  the  scale  described  earlier  to  gather  data  on  the  usability  and  utility  of  the  leader 
attribute  measures  (Appendix  A).  Note  that  the  leader  attributes  were  assessed  on  a  five-point 
scale;  for  this  evaluation,  the  mid-point  was  removed  to  promote  more  definitive  answers, 
resulting  in  a  four-point  scale. 

The  usability  and  utility  statements  were  based  on  heuristics  derived  from  research  by 
Nielsen  and  Mack  (1994)  and  Tognazzini  (2003)  that  represent  tool  usability.  Collectively,  these 
heuristics  or  design  principles  were  applied  to  develop  a  set  of  26  (ARC-FT)  and  22  (Integrated 
Database)  items.  These  items  were  structured  as  statements  that  were  framed  positively  or 
negatively  for  the  ARC-FT  and  Database,  respectively.  Participants  were  provided  a  document 
with  the  items  and  asked  to  check  agree  or  disagree  with  each  of  the  statements.  The  two  sets  of 
usability  statements  are  featured  in  Appendix  H.  These  statements  allow  for  a  more  fine-grained 
assessment  of  the  toolkit  by  asking  questions  about  specific  features,  as  opposed  to  the  more 
general  assessment  provided  by  the  standard  usability  questionnaire. 

The  interview  provided  an  opportunity  for  instructors  to  report  usability  issues  that  may 
not  have  been  captured  in  the  surveys.  In  particular,  the  interview  elicited  specific  feedback 
from  the  instructors’  experiences  in  using  the  tool  and  asked  the  user  to  elaborate  on  their 
responses  on  the  questionnaires.  The  interview  data  were  added  context  to  the  questionnaire 
results  and  identified  enhancements  to  the  tools.  One  interview  protocol  was  generated  to 
address  the  entire  toolkit  (see  Appendix  I). 
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Tool  evaluation  results.  Responses  to  the  usability  questionnaire  were  positive  on  all 
accounts.  For  the  supported  evaluation,  the  mean  rating  across  10  questions  was  3.65  ( SD=  .56) 
on  a  scale  of  4.  For  the  unsupported  evaluation,  the  mean  rating  for  the  FT  was  3.36  ( SD  =  .63). 
Finally,  for  the  unsupported  evaluation  for  the  Integrated  Database,  the  mean  rating  was  2.92 
(SD  =  .97).  Detailed  item-level  descriptive  statistics  for  each  portion  of  the  evaluation  are  in 
Appendix  J.  For  the  unsupported  evaluation,  the  items  with  the  lowest  ratings  were  those  that 
asked  about  the  complexity  of  the  system.  Based  on  the  ratings  from  the  usability  scale,  it 
appears  that  the  largest  challenge  with  the  FT  was  in  learning  the  system  and  becoming  familiar 
with  the  features  and  functionalities.  As  instructors  use  the  FT  more  often,  it  is  anticipated  that 
those  ratings  would  increase,  and  instructors  will  feel  more  comfortable  with  it.  As 
demonstrated  by  the  means,  the  instructors  reported  lower  ratings  for  the  database  compared  with 
the  FT.  As  expanded  upon  below,  the  instructors  did  not  use  the  database  as  much  as  the  FT,  and 
thus,  the  lower  ratings  are  not  completely  unexpected. 

Responses  to  the  usability  and  utility  statements  (Appendix  H)  were  largely  positive. 
Users  were  provided  with  two  options  for  responding  to  usability  statements  (Agree,  Disagree). 
Percent  agreement  was  calculated  and  reported  with  respect  to  Agree.  The  percent  agreement 
per  item  for  both  the  supported  and  unsupported  evaluations  is  contained  in  Appendix  J. 

Because  one  intent  of  using  this  measure  was  to  identify  specific  features  and  functionalities  that 
needed  to  be  changed,  the  research  team  set  a  strict  “passing”  criteria  -  80%  for  the  supported 
field  test,  and  75%  for  the  unsupported  leave-behind  for  the  FT;  anything  less  than  those 
standards  represented  a  feature  that  may  be  in  need  of  revision.  The  75-80%  criteria  were  set  so 
as  to  ensure  a  strict  level  of  scrutiny,  leading  to  the  identification  of  better  interactions  and 
interface  elements  in  the  final  development  iteration. 

For  the  supported  field  test,  20  out  of  26  items  reflected  the  passing  criteria  of  over  80% 
agreement.  For  the  unsupported  evaluation,  25  out  of  26  items  had  agreement  levels  of  75%  or 
above  for  the  FT.  In  both  cases,  the  failing  items  pointed  to  some  revisions  that  should  be  made 
to  the  tool.  For  example,  the  instructors  had  a  difficult  time  knowing  if  they  had  completed  an 
observation,  indicating  that  changes  to  the  interface  are  potentially  necessary.  In  both  the 
supported  and  unsupported  evaluations  for  the  FT,  users  were  not  always  aware  of  errors  made 
or  how  to  correct  those  errors. 

Users  did  not  embrace  the  Integrated  Database  as  enthusiastically  as  the  FT,  with  only  12 
out  of  22  items  receiving  “passing”  responses.  Two  caveats  exist  in  regard  to  these  data.  First, 
the  FT  was  more  frequently  used  by  instructors  than  the  database.  While  the  FT  was  carried  by 
the  instructors  throughout  the  course,  the  database  was  used  only  after  the  data  were  collected  on 
the  FT  by  each  rater.  Thus,  the  database  was  possibly  only  accessed  once  during  the  class. 
Further,  the  instructors  appeared  to  elect  an  unofficial  Database  Administrator  during  the  leave- 
behind  which  significantly  reduced  the  number  of  users.  This  point  leads  to  the  second  caveat, 
which  is  that,  given  the  low  response  rate,  further  testing  of  the  database  is  needed  to  better 
understand  its  strengths  and  weaknesses. 

Last,  instructors  briefly  interviewed  about  the  tool.  The  instructors  offered  a  number  of 
suggested  enhancements  (Table  3).  The  interview  data  mostly  represent  suggested 
enhancements  that  are  at  a  higher  level  than  the  more  specific  data  from  the  usability 
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questionnaires  (i.e.,  these  suggestions  represent  additional  features  versus  more  standard 
usability  issues). 

Table  3 

Qualitative  Feedback  for  ARC-PAT 


Link  with  Anny  network  and  shared  drives. 

Include  roll-up  data  of  student's  performance  by  phase 
Add  PDF  viewer  for  publications 

Discard  and  Close  should  be  deleted  or  changed  -  deletes  an  assessment 

Include  a  map  and  way  point  tracker 

More  in-depth  training  on  the  database 

Include  a  signature  pad  for  the  students  for  green/red  cards 

Implement  ability  to  drop  a  picture  in  an  assessment 

Green/red  card  stand-alone  functions  (print  independently  from  assessment) 

Develop  a  Print  function  within  the  field  tool 

Interface  between  app  and  photo  gallery  for  assessments 


Summary 

The  research  presented  here  describes  the  development  of  a  prototype,  proof-of-concept 
measurement  and  assessment  system.  The  ARC-PAT  was  developed  to  aid  instructors  in 
consistently  and  reliably  assessing  leader  attributes  and  to  provide  a  mechanism  for  tracking 
student  progress  between  and  within  instructors.  The  specific  tool  requirements  were  obtained 
from  experienced  ARC  instructors,  and  the  system  was  developed  to  meet  their  assessment 
needs.  More  generally,  however,  the  ARC -PAT  addresses  the  requirements  put  forth  by  the 
ALM  to  accurately  and  reliably  assess  21st  Century  Soldier  Competencies.  Without  accurate  and 
reliable  assessments,  training  cannot  successfully  be  tailored  to  the  individual  student. 

The  findings  from  the  initial  evaluations  are  favorable  and  indicate  that  the  ARC 
instructors  found  the  ARC-PAT  to  be  of  help  in  addressing  some  of  their  assessment  challenges. 
Although  several  modifications  and  enhancements  are  needed  for  the  tool  to  be  fully  functional 
within  the  course,  this  research  represents  a  step  forward  in  meeting  the  intent  of  the  ALM. 


12 


References 


Beaubien,  J.  M.,  Goodwin,  G.  F.,  Milanovich,  D.  A.,  Baker,  D.  P.,  &  Smith- Jentsch,  K.  (2004). 
Behavioral  observation  scales.  In  N.  Stanton,  A.  Hedge,  K.  Brookhuis,  E.  Salas,  &  H. 
Hendrick  (Eds.),  Handbook  of  human  factors  and  ergonomics  methods  (pp.  54-1  to  54-6). 
Boca  Raton,  FL:  CRC  Press. 


Bonnan,  W.  C.,  Hough,  L.  M.,  &  Dunnette,  M.  D.  (1976).  Development  of  behaviorally  based 
rating  scales  for  evaluating  U.S.  Navy  Recruiters  (Technical  Report  TR-76-31).  San 
Diego,  CA:  Navy  Personnel  Research  and  Development  Center. 

MacMillan,  J.,  Entin,  E.  B.,  Morley,  R.,  &  Bennett,  W.  (2013).  Measuring  team  perfonnance  in 
complex  and  dynamic  military  environments:  The  SPOTLITE  method.  Military 
Psychology,  25,  266-279. 

Nielsen,  J.,  &  Mack,  R.  L.  (1994).  Usability  Inspection  Methods.  New  York:  John  Wiley  &  Sons. 

Pulakos,  E.  (1997).  Ratings  of  job  performance.  In  D.  L.  Whetzel  &  G.  R.  Wheaton  (Eds.), 

Applied  Measurement  Methods  in  Industrial  Psychology,  (pp.  291-317).  Palo  Alto,  CA: 
Davies-Black  Publishing. 

Smith,  P.  C.,  &  Kendall,  L.  M.  (1963).  Retranslation  of  expectations:  An  approach  to  the 

construction  of  unambiguous  anchors  for  rating  scales.  Journal  of  Applied  Psychology, 
47,  149-155. 

Tognazzini,  B.  (2003).  http://www.asktog.com/basics/firstPrinciples.html.  Retrieved  June  12, 
2009. 

U.S.  Department  of  the  Anny,  Headquarters  (2011).  The  U.S.  Army  Learning  Concept  for  2015 
(TRADOC  PAM  525-8-2).  Fort  Monroe,  VA:  Author. 

U.S.  Department  of  the  Army,  Headquarters  (2012).  TRADOC  Quality  Assurance  (QA) 

Program  Self-Assessment  Guide  (Coordinating  Draft).  Fort  Eustis,  VA:  Author. 

U.S.  Department  of  the  Army,  Headquarters  (2013).  Army  Leader  Development  Program  (PAM 
350-58).  Washington,  DC:  Author. 


13 


APPENDIX  A 


Standard  Usability  and  Utility  Questionnaire  for  the  Leader  Attribute  Measures 


Directions:  Place  a  check  mark  in  the  box  that  best  represents  your  rating  of  your 
experience. 

Strongly  Strongly 

Disagree  Agree 


1 .  I  think  that  I  would  like  to  use  these  measures  and 
anchors  frequently. 

2.  I  found  the  anchors  to  be  unnecessarily  complex. 

3.  I  thought  the  anchors  were  easy  to  use. 

4.  I  thought  there  was  too  much  inconsistency  in 
anchors. 

5.  I  would  imagine  that  most  people  would  learn  to 
use  these  measures  and  anchors  very  quickly. 

6.  I  found  the  anchors  to  be  very  cumbersome  to  use. 

7.  I  felt  confident  using  the  anchors. 

8.  I  needed  to  learn  a  lot  of  things  before  I  could  get 
going  with  anchors  and  measures. 


A-l 


APPENDIX  B 


Item-Level  Descriptive  Statistics  for  the  Usability  and  Utility  Results  of  the  Leader 

Attribute  Measures 


Statement 

N 

Mean 

Median 

Mode 

1. 

I  think  that  I  would  like  to  use  these  measures  and 
anchors  frequently. 

5 

3.80 

4 

3 

2. 

I  found  the  anchors  to  be  unnecessarily  complex.* 

5 

1.60 

2 

2 

3. 

I  thought  the  anchors  were  easy  to  use. 

5 

4.00 

4 

4 

4. 

I  thought  there  was  too  much  inconsistency  in  the 
anchors.* 

5 

1.40 

1 

1 

5. 

I  would  imagine  that  most  people  would  leam  to  use 
these  measures  and  anchors  very  quickly. 

5 

4.40 

4 

4 

6. 

I  found  the  anchors  to  be  very  cumbersome  to  use.* 

5 

1.80 

2 

2 

7. 

I  felt  confident  using  the  anchors. 

5 

4.40 

4 

4 

8. 

I  needed  to  leam  a  lot  of  things  before  I  could  get  going 
with  the  anchors  and  measures.* 

5 

1.60 

1 

1 

*  These  items  were  reverse  coded  for  interpretation. 
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APPENDIX  C 


Final  Leader  Attribute  Measures 


STUDENT: 

CM: 

INSTRUCTOR: 

D4TE. 

ACCOUNTABILITY:  Takes  Responsibility  For  Own  And  Team’s  Actions  And  Consequences 


Delegates  Delegates  tasks  Appropriately  Communicates  Retains 

responsibility  without  follow-up  delegates  task/  information  responsibility 

without  oversight  authority  but  and  makes  and  verifies  that 

Does  not  with  limited  or  recommendations  delegated  tasking 

Mismanages  pass  critical  untimely  in  a  relevant  and  meets  mission 

timeline  and  information  in  a  follow-up  timely  manner  objectives  by 

misses  mission  timely  manner  engaging  in  a 

objectives  Communication  Mission  focused  timely  follow-up 

Task  focused  vs.  without  vs.  task  focused 

mission  focused  recommendation  (e.g.,  accounts  Makes  decisions 

for  follow  on  and  that  demonstrate 

Fails  to  take  Manages  time  to  adjacent  units;  understanding  of 

ownership  meet  minimal  commander's  mission  one  level 

of  mistakes/  mission  success  decision  making  up  and  two  levels 

miscalculations  criteria  (e.g.,  cycle)  down 

little/no  time  for 

rehearsals)  Expresses  intent 

and  takes 
ownership 
of  mistakes/ 
miscalculations 


CHUB/IBra 


. 

ADAPTABILITY: 

Manages  Changing  Requirements  For  Balancing  Unit  Recon,  Surveillance,  And  Security 

With  Mission  Accomplishment 

Inaction: 

paralyzed 


Rigidly  follows 
plan  in  changing 
tactical  conditions 
(e.g.,  “fights  the 
plan,  not  the 
battle”) 

Articulates  the 
problem  but  does 
not  work  toward  a 
solution 


Focuses  on  getting 
“right”  answer 

Articulates  the 
problem,  takes 
action,  but 
changes  are  not 
fully  effective 


Responds 
effectively,  but 
slowly  (may  need 
to  be  prompted  to 
stay  on  schedule) 

Adapts  principles/ 
doctrine  to 
attempt  to  solve  a 
new  problem. 


Changes 
behavior  in  a 
timely  manner 
in  response  to 
tactical  conditions 
to  effectively  meet 
purpose,  task,  and 
end  state 


C-l 


ANTICIPATION: 


Foresees  Future  Requirements  And  Conditions 


Constantly 
surprised  by 
unexpected 
consequences 


CBMOHS/iaiQ 


Sometimes  does 
not  anticipate 
unexpected  events; 
assumes  METT- 
TC  will  remain 
constant 

Reactive;  waits 
for  instructor 
directions/ 
prompts 


Links  tasks  to 
overall  mission 
with  guidance 

Focuses  on  local 
problem  (task  vs. 
mission  focused) 


Anticipates 
problems 
for  own  and 
follow  on  units; 
develops  multiple 
contingency  plans 


Stays  one 
step  ahead  of 
commander’s 
needs,  and 
identifies  second 
and  third  order 
effects 

Exploits 

opportunities  as 
they  emerge 


CONFIDENCE: 


Continuously 
seeks  input  from 
peers  without 
deciding/acting 

Fails  to  act  when 
challenged 


Believes  In  Own  And  Team’s  Ability  To  Handle  Tactical  Situations 


Runs  platoon  as  a 
democracy 

Changes  answers 
when  asked  “why;" 
chooses  not  to 
defend  logic 

Overly  confident; 
manages  all 
aspects  of 
planning  and 
execution  without 
accounting  for 
subordinate 
suggestions 


Incorporates  input 
from  subordinate 
leadership  with 
instructor  approval 

Task,  purpose, 
intent  is  over¬ 
complicated  or  too 
vague,  suggesting 
uncertainty 

Loses  momentum 
or  takes  too  much 
time  to  decide 


Considers 
recommendations 
from  subordinate 
leadership  and 
incorporates  as 
appropriate 

Provides  evidence 
when  asked  “why” 

Utilizes  the 
“strong”  students 
for  support  when 
necessary 

Effectively  conveys 
task,  purpose, 
intent  under 
moderate  levels  of 
stress 


Maintains 
command 
and  control 
under  difficult 
conditions  (e.g., 
extreme  stress, 
uncertainty)  and 
effectively  conveys 
task,  purpose, 
intent 
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INITIATIVE; 


Thinks  And  Arts  Without  Being  Urged 


Does  nothing,  even 
after  prompting 
from  the 
instructor 


Needs  to  be 
told  by  leader 
or  instructor  to 
execute  task 

Fills  white  space/ 
down  time  but  not 
effectively 

Does  not  capitalize 
on  known 
opportunities  to 
exploit  situations 


Seeks  approval 
for  ideas  prior  to 
acting  on  decision 

Takes  action 
without  fully 
accounting  for 
terrain,  threat, 
own  forces 
supported 

Takes  action  but 
is  unable  to  fully 
address  problem 
without  leader  or 
instructor  support 
(e.g.,  instructor 
provides  hint 
toward  solution) 


Typically  acts 
independently 
without  instructor 
support 

Effectively  uses 
white  space  to 
facilitate  mission 

Identifies  and 

incorporates 

multiple 

contingency  plans 


Identifies  and 
solves  problems  by 
maximizing  use 
of  enablers  and 
assets 

Continuously 
improves/ adjusts 
to  the  situation 

Proactively  exploits 
opportunities  to 
gain  additional 
information  or 
advantage  (e.g., 
switches  from 
reconnaissance 
to  surveillance  to 
security) 


.  pma  - 


J 


cmmvmB 


PROBLEM  SOLVING:  Solves  Problems  By  Applying  Deliberate  Thought 


Does  not  recognize 
when  problems 
arise 

Gives  up/does 
not  continue 
to  troubleshoot 
when  faced  with 
obstacles 


Identifies 
and  reacts  to 
immediate/local 
problems;  provides 
a  short-term 
solution 

Focuses  on  getting 
the  right  answer 
but  not  on  the 
process 


Recognizes 
problem  and 
identifies  one 
solution  within 
sphere  of  influence 
(e.g.,  uses  only 
organic  assets  to 
solve  problems) 

Uses  appropriate 
considerations 
(enemy,  terrain, 
infrastructure, 
society) in 
developing  COA 
when  guided  by 
instructor 


Employs  organic 
assets  and  plans 
for  the  use  of 
some  additional 
available  assets 

Applies  unique 
solutions  to  novel 
problems  but 
solution  may  not 
be  fully  effective 


Effectively 
employs  full  range 
of  available  assets 
and  capabilities 
and  incorporates 
all  relevant 
considerations 
(enemy,  terrain, 
infrastructure, 
society)  in 
developing  COA 

Consistently 
recognizes  and 
frames  problems 
and  works 
toward  a  range  of 
solutions  across 
multiple  problems 
with  no  instructor 
support 


COMB/MB 
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RISK  MANAGEMENT: 


Assesses  The  Situation  Against  The  Mission  And  Makes  A  Decision —  Effectively 
Balances  Mission  Requirements  And  Risk 


Rushes  to  solution 
without  taking 
proper  precautions 
or  developing 
contingency  plans 
(rushes  to  failure) 

Assumes  too  much 
risk,  or  over¬ 
mitigates  and  fails 
to  act 


Does  not  use 
available 
capabilities  to 
ensure  success 
and  minimize  risk 

Reactively 
manages  risk 
without  analysis 


Conducts  limited 
analysis  to  satisfy 
doctrine  or  tactical 
tasks  without 
understanding  of 
larger  mission, 
intent 

Integrates  risk 
mitigation  in  the 
planning  process 
but  does  not 
fully  execute  or 
make  in-stride 
adjustments 

Uses  available 
capabilities  to 
mitigate  risk  but 
with  instructor 
assist  ance 


-MTB- 


Plans  viable 
contingency  plans, 
and  executes  them 
as  needed 

Continuously 
plans  and 
communicates 
risk  mitigation  to 
the  appropriate 
organizational 
level  (i.e.,  lower, 
higher,  both) 

Uses  all  available 
capabilities  to 
mitigate  risk 


Conducts  cost- 
benefit  analysis, 
recognizes 
and  exploits 
opportunities/ 
enemy 
weaknesses, 
and  develops 
tactically  sound 
recommendation 
for  all  levels 
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APPENDIX  D 


Revised  Tool  Mock-Ups 


r 


Blackjack 


New  Observation 


9 


Blackjack 

Movement  to  point  B'avo 
Recce  target  K  b 

RELATED  FVFNTS:  FM  4-101:  FM  4-210 


nuary  16,2014 


A 


Select  a  student 


Select  Cav 


Select  event  and  choose  to  assess  or  rreasure 

O  Movement  to  point  Bravo 
O  Recce  target  Kilo 


10;S0  ▼  i 


fxt 


9 


Blackjack 

Movement  to  point  Bravo 
Recce  target  Kilo 


Thursday.  January  16,2014 


RELATED  EVENTS:  FM  4-101:  FM  4-210 


A 


Select  a  student 


CAV  2  Q 


£5  a 


Select  event  and  choose  to  assess  or  measure 

(•)  Movement  to  point  Bravo 
O  Recce  target  Kilo 


Assess 


1Q:SOtr. 


D-l 


Capture  performance  using  the  me 


Movement  to  contact 
Establish  Bivouac 


Date 

Thursday,  January  16,  2014 

Observation  Title 

Blackjack 

Accountability 


Adaptability 


0 


Demonstrate  ability  to  think,  persevere  under  physical  and  mental  pressure. 


CAV  2 

B 

&  Rated 

Vestiers.  E. 

2L.T 

Platoon  Leader 

jvf  Rated 

Smith,  F. 

SSG 

Gunner 

Cancel 


Save 
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00:01 :0  7 


o 


Lorem  ipsum  dolor  sil  arret,  consectetur  adiptscing  elit.  Nunc  ultricies  semper  dolor  rutrum  linciduni 
Suspend&se  ac  lempor  torlor  Suspendsse  x  portttor  melus  AJicuam  ultrcies  fcucibus  dolor,  vei  dictum 
iacus  linciduni  sit  arret  Proin  at  euismod  odio  Nulla  aiicuet  lortor  el  magna  ultamcorper,  nec  mterdum  libero 
consepuat  Nunc  veslibuium  dam  ut  pretium  bo  reel  erwn  justo  facilisis  rrassa  ut  pbceral  mi  iiguia  >  > 


4 


<4 


t 

« 

♦ 


Date 

Thursday,  January  16,  2014 

Observation  Title 

Blackjack 

Movement  to  contact 
Establish  Bivouac 


Accountability 


Adaptability 


Demonstrate  ability  to  think,  persevere  under  physical  and  mental  pressure 


CAV  2 


ft 


Cancel 
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Blackjack  >  Assess 

H  Thursday,  January  15,  2014 

Movement  to  contact 

Outcomes  A 


Save 


Attributes 


Vestiers,  E. 


Recommendations 


2LT 

Platoon  Leader 


B 


0* 

Observably  Higher  Fundamental  Skills 

® 

Attach 

•Exceed  Army  standards  for  land  navigation  tasks  and  skills. 

•Exceed  Army  standards  for  reporting  and  communicating  tasks. 

O 

/ 

•Demonstrate  improved  ability  to  assess  terrain,  enemy  capabilities, 
friendly  capabilities. 

© 

< 

•Demonstrate  ability  to  think,  persevere  under  physical  and  mental  pressure. 

Summary: 

© 

© 

Mission 

Per  fori 


o 

o 


O 

1Q:EOf. 

Blackjack  >  Assess 

ggj  Thursday,  January  15,  2014 


Save 


1 


A 


Movement  to  contact 


siu dent  Vestiers,  E.  2LT  Q 

Platoon  Leader 


Outcomes  A _ Attributes  A  Recommendations 


01  + 

Accountability  Takes  responsibility  for  own  and  team’s  actions 

PI  ©  (N.) 

Mission 

a 

and  consequences 

Performs 

Attach 

e 

•Dem 

Summary: 

@ 

? 

\ 

© 

A. 

W 

© 

o 

@ 

o 

oooooooooooo 


10:BD  ▼. 
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Blackjack  >  Trending 

[gj  Thursday,  January  16,  2014 

Performance  to  date 


Save 


H 


A 


Student 


Vestiers,  E.  2LT  Q 

Platoon  Leader 


Outcomes 


Accountability 


Anticipation 


+i 


Problem  Solving 


Risk  Management  / 
mitigation  (prudent  risk) 


-2 


Attributes 


Wednesday,  January  15,  2014 

Blackjack 

Movement  to  point 
Bravo 

Lorem  ipsum  do  b r  sit  amet, 
oonsectetur  adipecing  elit. 

Nunc  ultricies  semper  ctobr 
rutrum  tincidunt.  Suspendisse  ac  tempo  r 
tortor.  Suspendisse  ao porttitor  metus. 
Aliquam  ultricies faucibus dolor,  vel 
dictum  lacus  tincidunt  sit  amet.  Proin  at 
euismod  odio.  Nulla  aliquet  tortor  et  ullam- 
corper,  nec  interdum  libera...  More  ► 


2 


3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27 


1 is  | 

Phase  1 

Phase  II 

Phase  III 

+ 

LZ? 

1D:BD  ▼- 
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r 


Blackjack  >  Trending 


TO  Thursday,  January  16,  2014 

Performance  to  date 


Save 


s 


A 


Student 


Vestiers,  E.  2LT  Q 

Platoon  Leader 


Outcomes 


Initiative 


+i 


-2 


Attributes 


Wednesday,  January  15,  2014 

Blackjack 

Movement  to  point 
Bravo 

Lorem  ipsum  do  b  r  sit  amet, 
ronsectetur  adipecing  elit. 

Nunc  ultricies  semper  dobr 
rutrum  tincidunt.  Suspendisse  ac  tempo r 
tortor.  Suspendisse  ac  porttitor  metus. 
Aliquam  ultricies  faucibus  do  lor,  vel 
dictum  lacus  tincidunt  sit  amet.  Proin  at 
euismod  odio.  Nulla  aliquet  tortor  et  ullam- 
corper,  nec  interdum  libero...  More  ► 


Dave  1  2  3  4 

5  6 

7 

8  9  10  11  12  13  14 

15  16  17  18  19  20  21  22  23  24  25  26  27 

Phase  1 

✓ 

Phase 

II 

Phase  III 

+ 

1Q  =  5D  ▼, 
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APPENDIX  E 


Tool  User  Guide 


Army  Reconnaissance  Course  Performance 
Assessment  Toolkit  (ARC-PAT)  User  Guide 


APTIMA 

HUMAN-CENTERED 

ENGINEERING 


E-l 


Contents 


Introduction . E-3 

Support . E-3 

Overview . E-3 

Installation  and  Setup . E-4 

Operating  System  Requirements . E-4 

Software  and  Hardware  Requirements . E-4 

Installation . E-4 

Using  ARC-FT . E-ll 

Activity  1.  Database  Integration  and  Configuration . E-ll 

Activity  2.  Making  Observations . E-13 

Activity  3.  Completing  Assessments . E-15 

Activity  4.  Using  Trending  Displays . E-16 

Activity  5.  Assessment  Upload . E-17 

Integrated  Database . E-18 

Installation  and  Setup . E-18 

Operating  System  Requirements . E-18 

Software  and  Hardware  Requirements . E-18 

Installation . E-19 

Using  the  Integrated  Database . E-19 

Activity  1.  Startup  and  Login . E-19 

Activity  2.  Trend  Analysis . E-19 

Activity  3.  Documents . E-20 

Activity  4.  Administration . E-21 
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Introduction 


This  manual  provides  basic  user  guidance  for  the  ARC -Performance  Assessment  Toolkit. 
Details  about  the  purpose  of  the  tool  kit  and  the  individual  tools  are  included.  Should  this  text  be 
insufficient  to  resolve  technical  or  user  problems,  support  information  is  provided  below. 

Support 

Courtney  Dean 

781-496-2455 

cdean@aptima.com 

Overview 

The  Army  Reconnaissance  Course-Performance  Assessment  Toolkit  (ARC-PAT)  is  an 
integrated  mobile  data  collection  and  performance  assessment  system.  The  system  is  designed 
for  ARC  cadre  to  capture  and  rate  student  performance  as  they  progress  through  the  27-day 
course.  Data  can  be  collected  on  student  behaviors  reflecting  Expected  Outcomes  and  Leader 
Attributes  in  both  the  classroom  and  field  events.  ARC-FT  (ARC  Field  Tool)  is  a  digitized 
version  of  the  notebook  carried  by  cadre  and  the  paper  assessment  form  required  for 
documentation  of  student  progression.  The  ARC-FT  is  integrated  with  the  ARC  Integrated 
Database. 

The  Integrated  Database  is  a  web-based  software  that  stores  and  manages  student 
performance  data.  It  features  a  user  interface  that  supports  the  display  and  review  of 
performance  data  across  students,  units,  classes  and  even  years.  The  database  interface  offers 
tools  for  reviewing  the  data  to  identify  trends  and  patterns  that  will  help  instructors  and 
leadership  manage  the  course.  It  also  formats  student  performance  data  into  a  format  that  mirrors 
the  assessment  forms  used  by  cadre,  which  can  be  printed  and  stored  according  to  current  best 
practices.  It  is  hosted  on  a  desktop  and  offers  a  wireless  local  area  network  (LAN).  The  LAN 
allows  configurations  (e.g.,  student  rosters)  to  be  loaded  onto  the  field  tool  and  performance 
assessment  data  to  be  uploaded,  stored,  and  printed.  ARC-FT  enables  ARC  instructors  to  capture 
critical  performance  metrics  for  students  that  reflect  learning  and  progression  within  the  ARC.  It 
supports  multi-media  capture  of  student  behavior  including  photo,  video  and  voice-text  and 
allows  users  to  attach  these  data  to  Leader  Attributes  and  Expected  Outcomes,  reducing  repeated 
efforts.  These  captured  behaviors  can  later  be  attached  to  the  assessments  of  those  same 
outcomes  and  attributes. 

This  manual  describes  how  to  install  and  use  ARC-FT  and  the  ARC  Integrated  Database. 
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ARC-FT 


Installation  and  Setup 

This  section  details  the  steps  necessary  to  install  ARC-FT.  These  instructions  are  based 
on  the  assumption  that  the  user  has  administrative  privileges  on  the  device  on  which  the  software 
will  be  installed.  The  software  has  been  designed  to  run  on  a  mobile  device  running  the  Android 
Operating  System. 


Operating  System  Requirements 

ARC-FT  has  been  tested  on  Android  operating  systems  (4.1  and  up).  Installation  on 
other  operating  systems  is  currently  not  supported  and  is  done  at  the  user’s  own  risk. 

Software  and  Hardware  Requirements 

•  8+  inch  Android  Tablet 

•  1  GHz  processor  minimum 

•  400MB  RAM 

•  2  GB  storage  space  minimum  (4GB  recommended) 

•  ARC-FT  requires  Android  4.1  or  higher. 

•  ARC-FT  requires  10  MB  of  internal  storage. 

Installation 

This  section  covers  the  steps  necessary  to  install  ARC-FT  on  an  Android  device.  These 
instructions  are  based  on  the  assumption  that  the  user  has  access  to  an  Android  v4. 1  or  higher 
tablet  or  smartphone  and  a  connecting  USB  cable.  The  steps  covered  in  this  section  are 
consistent  for  all  versions  of  ARC-FT. 

Step  1.  Begin  Installation 

1 .  Access  arcft.apk  on  your  PC 

a.  This  may  have  been  received  via  email,  DVD/CD  ROM,  or  FTP 

b.  Save  the  .apk  fde  to  your  desktop  (or  a  designated  folder  on  the  PC) 

2.  Connect  tablet  to  computer  via  USB  cable 

3.  When  Windows  prompts  user  to  act  on  the  device,  select  “Open  device  to  view  files” 
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4. 


AutoPlay 


4 


GT-N5110 


^[Always  do  this  for  this  device: 

Device  options 

6 


Sync  digital  media  files  to  this  device 

using  Windows  Media  Player 


k 


Open  device  to  view  files 

using  Windows  Explorer 

Import  pictures  and  videos 

using  Windows 


View  more  AutoPlay  options  in  Control  Panel 


Double-click  on  Tablet 


VT  Favorites 
K  Desktop 
JO-  Downloads 
%  decent  Places 

a  Libraries 
H  Documents 
J  Music 
IP  Pictures 
3  Videos 

iflU  Comouter 


Tablet 

8.92  GB  free  of  9.67  GB 
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5.  Open  the  “Download”  folder  on  the  Android  Tablet 


Computer 
Windows  (C:) 
i  i  Seagate (E) 
,j£  GT-N5110 
„  Tablet 

^1  Network 


Pictures 
File  folder 


Ringtones 
File  folder 


ScreenSaver 

File  folder 


Playlists 
File  folder 


Date  created:  7/30/2014  9:38  AM 
Notifications  |  Date  modified:  7/30/2014  9:38  AM 
File  folder 


Podcasts 
Ele  folder 


* 


S  Note 
File  folder 


Samsung 
File  folder 


X 


Download  Date  created:  7/30/2014  9:38  AM 
File  folder  Date  modified:  7/30/2014  9:38  AM 
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Step  2.  Copy  Installation  File 

1 .  Connect  your  Android  Mobile  device  to  your  PC. 

2.  Locate  the  ARC-FT  installer  file  on  your  PC.  Note  the  title  will  likely  be  “ArcFt” 
followed  by  a  version  number.  What  is  important  is  that  the  file  type  is  “.apk,”  which  is  a 
mobile  app  installer. 

3.  Right  click  the  file  and  select  “Copy” 


4.  In  the  white  space  of  the  Download  folder  on  the  Tablet  window  right  click  and  select 
“Paste” 
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Step  3.  Access  the  Android  Device’s  File  Manager 

1 .  Disconnect  the  tablet  from  the  PC  by  removing  the  USB  cable  from  the  Tablet 

2.  Open  the  Android  device’s  Application  Menu  to  access  the  File  Manager.  If  the  device 
does  not  come  with  a  pre-installed  File  Manager,  many  options  exist  in  the  Google  Play 
Store  {Astro  File  Manager  is  recommended). 


3.  Select  the  “My  Files”  app 


jontacl 


Game  Hub 


Play  Books  Play  Mag. 


Music  Pla. 


Navigation 


Apps  .i^|^^^Widgets 

"^larm^^^^|^^otejH^^|^Calculator  Camera  ChatON  Chrome 


)<«  »  :  9:20  AM 

* 


Flipboard 


Google  S. 


Group  Play 


Maps  Messenger  Music  Hub 


Internet 
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Step  4.  Locate  the  ARC-FT  Installation  File 

The  ARC-FT  installation  file  can  be  found  in  the  same  folder  you  pasted  the  .apk  into  (usually 
“Download”).  If  it  was  copied  to  a  different  directory,  that  folder  can  also  be  accessed  through 
the  File  Manager. 

1 .  Navigate  to  the  “Download”  folder 


My  Files 


Q,  :: 


/storage/sdcardO 


Time 


storage 

a 

■ 

Download 

07/30/2014  9:38  AM 

sdcardO 

_  Alarms 

+  Android 

□ 

■ 

aptima 

07/07/201412:22  PM 

□ 

m 

Pictures 

04/07/2014  1:39  PM 

+  aptima 

□ 

m 

Playlists 

06/14/2013  8:27  PM 

|  DCIM 

□ 

M 

Notifications 

12/31/2012  7:02  PM 

m  Documents 

□ 

■ 

Screensaver 

12/31/2012  7:02  PM 

2. 


The  open  Download  folder  should  show  the  ARC-FT  .apk.  Select  it  and  follow  the 
instructions  on  the  pop-up  windows.  Note:  you  may  get  a  warning  and  a  request  to 
authorize  installation  of  a  file  from  an  unknown  source.  OK  this  process.  ARC-FT  is 
tested  thoroughly  and  contains  no  nefarious  code. 


/storage/sdc...O/Download 
storage 
r  sdcardO 
Alarms 
+  Android 


■  n 

Download 

|  Time  ~A: 

□ 

m 

%  ArcFt  1.0.2.4.apk 

6.1  MB 

10/01/2014  5:00  PM 

□ 

■  _  **  «■ 

%  ArcFt  1.0.2.2.apk 

6.1  MB 

08/27/2014  10:05  PM 

□ 

n 

%  ArcFt  1 .0.2.1. apk 

6.1  MB 

i  08/27/2014  10:05  PM 

DCIM 


Documents 

ll  Download 
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Step  5.  Install  ARC-FT 

Open  the  ARC-FT  installation  file  in  the  File  Manager  and  press  Install.  If  you  already  have  the 
application  on  your  tablet,  you  will  get  the  following  warning:  Press  OK  to  update  your  app  to 
the  new  version.  If  you  do  not  already  have  an  older  version  installed,  this  message  will  not 
display. 

1 .  Click  “Install”  from  the  pop-up  window. 


The  application  will  install 
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Step  6.  Complete  Installation 

A  confirmation  message  will  appear  when  installation  is  complete.  Select  Done  to  exit  setup  and 
touch  the  Home  Screen  icon  to  exit  the  file  manager.  Selecting  “Open”  will  open  the  application. 


Using  ARC-FT 

There  are  five  primary  activities  in  ARC-FT : 

1 .  Database  Integration  and  Configuration 

2.  Observations 

3.  Assessments 

4.  Trending  Displays 

5.  Assessment  Upload 

This  guide  will  discuss  the  procedures  required  to  accomplish  all  of  these  activities  in 
ARC-FT. 

Activity  1.  Database  Integration  and  Configuration 

1 .  The  first  step  in  using  ARC-FT  is  to  connect  the  mobile  application  to  the  ARC 
Integrated  Database.  Note  that  this  should  be  done  for  you  prior  to  distribution  of  the 
devices.  However,  it  may  be  that  not  all  tablets  were  networked  or  that  an  has  reset  these 
settings.  To  connect  ARC-FT  to  the  database,  the  Android  device  must  be  connected  to 
the  ARC  local  area  network  (LAN).  The  wireless  router  supports  a  limited  range  of  a  few 
hundred  feet.  Thus,  the  integration  and  configuration  will  likely  have  to  occur  at  Patton 


E-ll 


Hall  (or  wherever  the  database  is  kept)  prior  to  taking  ARC -FT  into  the  field.  To  connect 
the  device  to  the  LAN,  pull  up  the  Android  Device’s  “Settings”  application. 


2. 


From  the  Settings  application,  select  the  “Connections”  tab  and  turn  on  the  “Wi-Fi” 
switch.  Search  the  wireless  connections  for  “TP-LINK_Pocket_3020_947F18”.  Tap 
ARC-FT  to  connect.  You  will  be  prompted  to  provide  a  Network  Key.  The  key  is 


13372903. 


3. 


Once  you  are  connected,  you  must  test  the  connection.  Open  the  internet  browser  ! 
and  type  “http://l  92. 168.0. 100/ari”.  If  the  tablet  is  connected,  then  the  following  website 
should  appear. 


•  □  a  #  v  £  S  n  ¥  c  1 


*  a  a 

•  D  2 


ASA  System  Status 
ASA  Server  Version:  1.1. 1.0 


Data  Model  Version:  1.1.0.64-vll 
Database  Name:  ASA_Server 
Web  API  Controllers: 


Name:  HpmlModule 
Version;  i.«.o.iS6 
HPMLContioIler 

Same:  SpoiliteModnle 
N'uGet  Version;  11.0.164-vu 

Spotlit 'Controller 


Same:  System. Web. Http.ODati 


Registered  ASA  Modules: 


To  start  ARC-FT,  tap  the  ARC-FT  icon 


in  the  apps  menu. 


4.  The  final  step  in  connecting  to  the  database  is  to  select  the  Menu  in  the  top  right  corner  of 
the  app  interface.  From  this  menu,  select  “Settings.” 


5.  From  the  Settings  there  is  a  space  to  type  in  the  “ASA  Server  URL.”  The  URL  should 
read  “http:// 1 92. 168.0. 100/ari.”  If  it  does  not,  then  replace  the  text  with  that.  Tap  Sync 
Configurations  with  ASA. 


6.  Tap  “Pick  Configuration.”  A  list  of  configurations  should  appear.  These  should  be 

formatted  like  the  ARC  course  structure,  year  and  class  number  (e.g.,  14-008).  Select  the 
appropriate  configuration  (likely  the  latest).  This  will  load  the  application  with  the 
student  roster. 


7.  From  the  top  menu,  again,  select  the  instructor  option  (should  be  the  first  item  on  the 
menu).  A  profile  menu  will  appear.  Fill  out  title,  first  and  last  name.  Your  name  will 
now  appear  in  the  menu  next  to  the  instructor  icon. 
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Activity  2.  Making  Observations 

The  Observations  function  allows  instructors  to  perfonn  opportunistic  data  collection. 
This  set  of  tools  is  designed  to  provide  multi-media  options  for  capturing  notes  about  student 
behavior  quickly  and  at  the  time  of  occurrence.  Instructors  using  ARC-FT  should  be  able  to  pull 
out  the  tablet  and  either  photo/video  student  behavior,  or  dictate/type  descriptions  of  what  was 
observed  and  associate  the  photo,  video,  or  text  with  the  student  and  the  Expected  Outcomes  and 
Leader  Attributes  that  are  key  indicators  of  student  progression  in  the  course. 

1 .  At  the  start  of  the  training  day,  the  ARC-FT  tool  will  need  to  be  set  to  the  proper  day  and 
event.  Tap  the  “Training  Day”  calendar  icon  in  the  top  middle  of  the  interface  to  set  the 
training  day. 

2.  Select  the  training  event  near  the  top  right  of  the  interface.  The  dropdown  menu  icon  will 
open  up  the  list  of  events. 
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Cav  1 

Student  1  2LT  Platoon  Leader 
Student  2  SSG  Gunner 
Student  3  2LT  Dismount 
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i 

Area  Reconnaissance  and  OP  STX 

Bushmaster 

Communications  and  Sensors 

Continuous  Reconnaissance 

Ergo 

IPB  for  Reconnaissance  and  Security 

Inprocessing 

LZ-PZ  Ops 

Operation  Golden  Eye 

3.  The  training  day  and  event  should  be  addressed  every  day  to  ensure  the  student 
assessments  are  captured  for  the  correct  events. 

4.  To  capture  an  observation,  select  any  of  the  “New  Observation”  buttons  that  appear  in  the 
top  left  corner  of  the  interface.  There  are  three  options:  Camera,  Voice  to  Text,  Notes. 

a.  Camera  allows  the  user  to  capture  photo  or  video  of  the  student(s) 

b.  Voice  to  text  allows  the  instructor  to  dictate  his/her  observations  and  the  device 
will  transcribe  the  speech. 

c.  Notes  allow  the  instructor  to  type  a  narrative  description  of  the  student(s) 
behavior. 


5.  After  capturing  a  photo,  or  video,  the  tool  will  provide  a  supplemental  data  screen.  Voice 
to  Text  and  Notes  will  begin  with  this  screen.  This  screen  will  prompt  the  user  to  provide 
additional  information  about  the  observation.  This  includes  a  type-written  summary 
description,  checklists  of  Expect  Outcomes,  Leader  Attributes  and  the  students  in  the 
current  class,  organized  by  Cav. 

6.  To  provide  a  summary,  tap  the  text  box  under  the  Summary  header.  The  summary  can 
later  be  pasted  into  the  assessment  form  so  it  is  important  to  provide  meaningful 
information  here  and  to  select  the  appropriate  Attributes  and  Outcomes  and  link  the 
observation  to  the  appropriate  student. 

7.  To  select  Attributes  and  Outcomes,  select  the  &  icon  to  the  right. 
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8.  To  select  one  or  more  students,  tap  the  check  box  to  the  left  of  each  student’s  name. 

9.  A  Green  Card  icon  is  also  made  available  next  to  each  student’s  name  -  1  to  allow 
instructors  to  document  the  issuing  of  a  green  car  if  it  occurred.  Tapping  on  the  green 
card  icon  will  change  it  from  gray  to  green.  If  the  green  card  was  activated  in  error, 
simply  tap  it  again  to  return  the  icon  to  gray. 

10.  At  the  bottom  of  the  interface,  Cancel  and  Save  buttons  are  available  to  complete  the 
observation.  Cancel  will  delete  any  record  of  the  observation.  This  should  only  be  used  if 
the  observation  was  started  in  error. 

Activity  3.  Completing  Assessments 

The  Assessment  function  is  a  digital  version  of  the  assessment  form  completed  by 
instructors  to  document  student  progress.  The  Assessment  function  enables  users  to  paste 
summary  comments  from  observations  eliminating  redundant  steps.  Completed  assessments  are 
also  uploaded  from  the  ARC -FT  to  the  Integrated  Database  where  they  are  stored,  reviewed,  and 
printed. 

1 .  To  begin  an  assessment,  select  a  student  from  the  list  of  students  in  the  bottom  left  half  of 
the  ARC-FT  interface.  The  students  are  organized  by  Cav.  To  navigate  the  list  of  Cavs, 
expand  the  dropdown  menu  by  tapping  on  the  gray  triangle  in  the  comer.  Select  the 
appropriate  Cav  and  then  tap  the  student  name. 

a.  An  optional  Leadership  Role  selection  for  students  is  included  on  this  list.  If  a 
student  is  performing  in  a  leadership  role  (e.g.,  Platoon  Sergeant,  Alpha  Team 
Leader)  that  role  can  be  specified  by  tapping  on  the  down  facing  arrow  next  to  the 
Student’s  name.  The  selected  role  will  then  be  listed  on  the  assessment  form  (in 
print  form).  If  no  role  is  selected,  then  that  entry  will  be  left  blank  when  the  fonn 
is  printed. 

2.  Once  a  student  is  selected,  tap  the  “Assess”  button  to  open  the  Assessment  screen.  The 
Assessment  screen  will  be  characterized  by  the  Event  in  the  top  left  comer,  the  date 
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below  that,  and  the  student’s  name  on  the  right  side.  It  features  three  tabs:  Outcomes, 
Attributes  and  Recommendations. 

3.  Each  Outcome  and  Attribute  can  be  rated  from  -2  -  +2,  or  NA  if  no  rating  is  made.  A 
summary  blank  is  provided  to  justify  the  score.  The  attachments  icon  on  the  left  side  of 
the  interface  allows  the  user  to  attach  any  summary  descriptions  provided  in  the 
observations. 

4.  To  select  different  outcomes  or  attributes,  tap  the  left  and  right  toggle  switches.  These 
will  cycle  through  the  list  of  outcomes  or  attributes. 


5.  To  attach  an  observation,  select  the  Attachments  icon.  A  pop-up  will  appear  listing  the 
relevant  observations.  Only  observations  that  are  linked  to  the  currently  selected 
Outcome  or  Attribute  will  appear.  Observations  are  characterized  by  their  type  with  an 
icon  displayed,  by  their  name,  and  usually  the  date  and  event.  Select  one  or  more 
observations  by  tapping  on  the  checkbox  to  the  left  of  the  attachment  icon.  Attaching  an 
observation  will  paste  any  summary  text  into  the  summary  blank  on  the  assessment  page. 
Note  that  this  must  be  done  for  each  Attribute  and  Outcome  separately. 

6.  Leader  Attributes  have  an  additional  data  input  option.  The  PI  button  allows  instructors 
to  review  a  set  of  performance  indicators  that  characterize  observable  behaviors 
demonstrated  by  students.  These  can  be  selected  by  instructors  to  add  additional 
information  to  the  assessment.  To  select  a  performance  indicator,  check  the  box  to  the  left 
of  the  phrase. 

Activity  4.  Using  Trending  Displays 

The  Trending  Display  features  a  line  graph  for  each  student  with  nodes  representing  the 
ratings  captured  for  Attributes  and  Outcomes.  It  allows  instructors  to  see  student  progress  in  a 
high  level  graphical  fonn  that  is  easily  interpreted,  yet  offers  drill-down  options  with  detailed 
information. 

1 .  To  access  the  Trending  Display,  tap  the  trending  button  in  the  top  right  comer  of  the 
interface  to  access  the  Trending  Display. 


E-16 


2.  The  Trending  Display  will  open  showing  aggregate  data  for  the  selected  student.  NOTE: 
The  tool  will  default  to  the  first  student  in  the  roster  list  and  Expected  Outcomes  for  that 
student.  The  graph  will  feature  an  individual,  color-coded  line  for  each  Expected 
Outcome.  Each  line  will  have  one  or  more  circular  nodes  that  align  to  the  performance 
scale  (e.g.,  +2  -  -2)  on  the  vertical  axis  (Y)  and  the  training  day  (e.g.,  1-27)  it  was 
collected  on  the  horizontal  axis  (X). 

3.  If  the  graph  is  too  cluttered,  individual  Outcomes  can  be  hidden  from  view.  The 
Outcomes  are  listed  with  their  representative  colors  in  the  window  to  the  left  of  the  graph. 
Tapping  on  any  of  these  Outcomes  will  hide  it  from  view.  The  Outcome  name  will  be 
faded  in  the  window.  To  show  the  line  graph  for  a  hidden  Outcome,  simply  tap  the  name 
of  the  outcome  again. 

4.  Each  Node  on  the  graph  represents  an  assessment  made  for  that  Outcome.  Tap  the  node 
to  see  additional  information  about  the  assessment,  including  the  summary  narrative,  date 
of  collection,  and  the  training  event.  A  pop-up  will  appear  displaying  these  data.  The 
pop-up  will  be  proximal  to  the  node  it  represents  and  will  feature  a  like-colored  border. 
These  features  help  the  user  ensure  that  the  information  displayed  represent  the  desired 
node,  which  is  critical  when  many  ratings  have  been  made. 

5.  All  of  the  features  described  in  3-5  are  available  for  Leader  Attributes.  To  view  results 
for  Attributes,  simply  tap  the  Attributes  tab  above  the  graph.  Individual  Attributes  will  be 
represented  by  color  coded  lines  and  nodes  that  can  be  shown  or  hidden  and  opened  to 
view  supporting  infonnation. 

6.  There  is  a  student  identifier  and  dropdown  list  in  the  trending  display.  This  allows 
instructors  to  see  which  student  is  selected,  verify  that  the  correct  student’s  data  are 
displayed,  and  to  change  students.  To  select  a  different  student,  select  the  dropdown 
arrow  to  the  right  of  the  student’s  name. 

Activity  5.  Assessment  Upload 

ARC-FT  enables  trainers  to  push  data  collected  in  the  field  to  the  database  for  storage 
and  examination.  Uploading  data  to  the  database  functions  much  like  integration  and 
configuration.  The  tablet  must  be  connected  to  the  local  area  network  and  the  application  should 
be  connected  to  the  database  through  that  wireless  connection.  It  is  recommended  that  instructors 
upload  data  frequently  to  minimize  loss  due  to  damage  of  tablets  and  to  keep  the  database 
current. 

1 .  Connect  the  mobile  application  to  the  ARC  Integrated  Database.  Note  that  this  should  be 
done  for  you  prior  to  distribution  of  the  devices.  However,  it  may  be  that  not  all  tablets 
were  networked  or  that  an  update  has  reset  these  settings.  To  connect  ARC-FT  to  the 
database,  the  Android  device  must  be  connected  to  the  ARC  local  area  network  (LAN). 
The  wireless  router  supports  a  limited  range  of  a  few  hundred  feet.  Thus,  the  integration 
and  configuration  will  likely  have  to  occur  at  Patton  Hall  (or  wherever  the  database  is 
kept)  prior  to  taking  ARC-FT  into  the  field.  To  connect  the  device  to  the  LAN,  pull  up 
the  Android  Device’s  Settings  application. 
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2.  From  the  Settings  application,  select  the  Connections  tab  and  turn  on  the  Wi-Fi  switch. 
Search  the  wireless  connections  for  “TP-LINK_Pocket_3020_947F18”.  Tap  ARC-FT  to 
connect.  If  the  tablet  has  been  previously  connected  to  the  network,  the  network  key 
should  be  known  by  the  device. 

3.  Once  you  are  connected,  you  must  test  the  connection.  Open  the  internet  browser  and 
type  “http:// 1 92. 168.0. 100/ari”.  If  the  tablet  is  connected,  then  the  following  website 
should  appear. 

Once  a  connection  is  confirmed,  return  to  the  ARC-FT  app. 

4.  Select  Settings  from  the  top  menu. 

5.  A  pop-up  will  appear.  Press  the  “Sync  Configurations  with  ASA”  button. 

6.  A  pop-up  will  appear.  Confirm  that  the  “ASA  Server  URL”  is  correct  at 
“http://192. 168.0. 100/ari” 

7.  If  the  URL  is  correct,  Press  “Sync  Configurations  with  ASA. 

8.  Press  Done. 

9.  Select  Export  option  from  the  top  menu.  This  will  up  now  upload  the  data  collected  for 
ARC  students. 

Integrated  Database 


Installation  and  Setup 

This  section  details  the  system  requirements  and  setup  of  the  Integrated  Database.  Note 
that  installation  instructions  are  provided  in  a  separate  set  of  documents.  These  instructions  are 
based  on  the  assumption  that  the  user  has  administrative  privileges  on  the  device  on  which  the 
software  will  be  installed. 


Operating  System  Requirements 

The  Integrated  Database  is  compatible  with  Windows  7  or  higher,  but  requires  64-bit 
operating  system. 


Software  and  Hardware  Requirements 

•  Desktop  or  Laptop 

•  2.9  GB  or  higher  processor 

•  8  GB  of  RAM 

•  Windows  7 

•  64-bit  Operating  System 

•  1 50Mbps  or  higher  Wireless  Router 

•  Google  Chrome  (Recommended) 
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Installation 


The  Integrated  Database  should  come  installed  and  ready  to  use.  It  is  not  recommended 
that  this  be  removed  or  modified  as  the  installation  process  is  a  complicated  process  and  is  not 
recommended  without  the  assistance  of  a  software  engineer. 


Using  the  Integrated  Database 

There  are  four  primary  activities  in  the  Integrated  Database 

1 .  Startup  and  Login 

2.  Trend  Analysis 

3.  Documents 

4.  Administration 

This  guide  will  discuss  the  procedures  required  to  accomplish  all  of  these  activities  in  the 
Integrated  Database. 


Activity  1.  Startup  and  Login 

The  integrated  database  can  be  accessed  through  a  web  browser.  The  interface  has  been 
optimized  for  Google  Chrome  so  it  is  recommended  that  the  user  use  Chrome  unless  the  browser 
software  is  not  available.  Microsoft  Internet  Explorer  version  9  or  higher  will  also  work. 

1 .  Connect  the  wireless  router  to  the  PC  or  Laptop.  Insert  one  end  of  the  CAT5  (Ethernet) 
cable  into  the  computer  and  the  other  end  into  the  router.  Make  sure  to  power  the  router 
either  by  the  wall  plug  or  with  the  USB  cable.  Also,  ensure  that  the  router  is  switched  to 
WISP. 

2.  Open  Google  Chrome  and  type  the  following  URL  into  the  search  menu: 

“http://192. 168.0. 100/ari” 

3.  The  browser  should  navigate  to  the  Database  home  screen.  This  screen  will  be  primarily 
white,  with  no  content  except  for  the  system  name  in  the  top  left  comer  and  a  login 
button  on  the  right  comer.  Because  there  is  sensitive  personnel  information  contained  and 
displayed  in  this  system,  access  is  denied  without  a  proper  login. 

4.  To  login,  tap  the  Login  button.  A  pop-up  will  provide  inputs  for  username  and  password. 
The  default  username  and  password  for  the  system  are: 

Username:  administrator 

Password:  tester 

5.  Once  logged  in,  menu  items  will  appear  in  the  top  menu.  There  are  three:  Trend  Analysis, 
Documents,  and  Administration. 

Activity  2.  Trend  Analysis 

The  Trend  Analysis  tools  offered  in  the  Integrated  Database  go  beyond  what  is  offered  in 
the  Mobile  Application.  These  tools  allow  instmctors  and  course  administrators  to  view  multiple 


E-19 


students  at  the  same  time,  and  compare  group  averages  at  the  CAV,  Class,  or  even  Year  level. 
With  the  Integrated  Database,  all  of  the  collected  and  uploaded  data  are  accessible. 

1 .  Select  the  Trend  Analysis  icon  in  the  top  menu. 

2.  A  set  of  filters  will  appear  on  the  left  hand  side  of  the  interface.  The  top  filters  are: 
Students,  Instructors,  and  Events.  Each  of  these  top  filters  will  organize  the  data 
accordingly  and  open  up  additional  filters  to  allow  users  to  dive  deeper. 

NOTE:  Currently,  the  only  high  level  filter  available  is  the  student  view.  The 
instructor  and  Even  ts  filters  are  still  in  developmen  t. 

3.  Select  the  student  filter  and  a  second  set  of  filters  appears.  These  filters  enable  users  to 
look  at  student  data  by  year,  class,  Cav  or  individual.  The  values  in  each  of  these  filters 
are  multi-select.  Therefore  a  user  can  select  two  or  more  years,  courses,  Cavs  or  students 
for  comparison.  Note  however,  that  if  two  or  more  items  are  selected,  the  next  level  down 
will  be  disabled.  In  other  words,  if  two  Cavs  are  selected,  then  the  data  displayed  will  be 
an  average  of  student  scores  from  those  two  Cavs  in  two  sets  of  columns  (one  column  per 
Outcome,  or  Attribute). 

4.  In  the  example  below,  three  students  from  year  2014,  class  14-100a  (a  fictional  course), 
Cav  1  were  selected.  Two  of  these  students  have  data.  The  third  does  not.  Average 
ratings  for  three  different  Expected  Outcomes  appear  based  on  the  data  in  the  database 
for  each  of  the  two  students. 

5.  Graph  displays  will  default  to  Expected  Outcomes.  To  view  Attribute  data,  select  the 
“Attributes”  tab  above  the  graphs. 

6.  Different  filtering  combinations  will  yield  different  data  displays.  As  more  data  are 
entered,  more  interesting  and  useful  trending  data  will  be  made  available. 

Activity  3.  Documents 

The  Integrated  Database  supports  current  best  documentation  practices  within  the  ARC. 
The  software  produces  Assessment  forms  that  should  mirror  those  currently  used  by  cadre  for 
students.  These  fonns  can  then  be  printed  and  stored  as  per  current  practice. 

1 .  Select  the  Documents  option  from  the  top  menu. 

2.  A  set  of  menu  items  should  appear  on  the  left  hand  side  of  the  interface.  The  View 
Assessments  option  accesses  the  completed  and  uploaded  student  assessment  fonns. 

3.  Student  fonns  are  accessed  using  the  same  filtering  system  offered  in  the  Trend  Analysis 
function.  Cadre  can  select  one  or  more  years,  classes,  Cavs  or  students  to  view  and  print 
forms.  The  filters  work  in  the  same  way  so  if  multiple  students  from  a  Cav  are  selected, 
then  forms  for  each  student  will  appear. 

4.  Once  selections  have  been  made  in  the  filters,  the  completed  Assessment  forms  will 
appear  in  the  middle  of  the  interface.  The  forms  are  not  editable  in  this  view.  However,  if 
data  are  missing  (such  as  duty  position  or  rating  instructor)  once  printed,  they  can  be 
modified  with  pen  or  pencil. 

5.  To  print  the  fonn,  select  the  Print  Assessment,  or  Print  Assessment  Group  buttons  above 
the  fonn.  The  Print  Assessment  button  will  print  only  one  assessment  at  a  time.  The  Print 
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Assessment  Group  button  will  print  all  forms  from  within  the  filtering  criteria  (e.g.,  all 
forms  for  selected  student,  or  Cav). 

6.  The  Raw  Data  Manipulation  is  the  second  menu  item  on  the  left  hand  side.  This  is 
primarily  a  developer’s  tool.  It  provides  a  view  of  the  data  captured  and  some  tools  to 
modify  those  data,  but  is  currently  not  user  friendly.  It  is  highly  recommended  that  this 
function  be  ignored  while  the  tool  remains  in  development. 

Activity  4.  Administration 

The  Administration  features  provides  access  and  data  management  tools  for  leadership 
and  cadre.  This  menu  item  features  Roster  Upload  and  management,  Instructor  Profiles  and  User 
Access  controls.  The  administration  functions  can  enable  restrictions  to  be  made  on  who  can  see, 
or  manipulate  data  in  the  database.  For  purposes  of  maintaining  sensitive  data  integrity  and 
management  of  the  system,  it  is  recommended  that  someone  be  assigned  administrator  of  the 
Integrated  Database  who  is  responsible  for  granting  access  to  the  tool. 

7.  The  administration  button  is  featured  on  the  top  menu.  Selecting  it  opens  up  several 
controls  on  the  left  hand  side.  These  include  User  Administration,  User  Access  Control, 
Class  Roster,  and  Instructor  Roster 

8.  User  Administration  allows  an  administrator  to  add  users  (generally  course  instructors 
and  leadership).  Select  Add  User  and  complete  the  form  for  the  new  user.  Select  the 
appropriate  role  for  the  user  based  on  the  rights  they  should  have.  There  should  only  be 
one  Administrator.  Managers  have  read  and  write  privileges,  but  cannot  control  other 
users’  access.  Users  have  much  more  limited  controls.  Most  users  should  probably  have 
Manager  credentials. 

Note:  The  user  list  provides  access  to  all  users  and  roles  can  be  changed  there  if  the  need 
arises.  The  user  list  features  all  of  the  Administrators,  Managers  and  Users  with 
credentials  currently  in  the  system. 

9.  User  Access  Control  is  a  detailed  administrative  function.  This  function  allows  an 
Administrator  to  specify  the  data  a  user  can  access,  view,  and/or  manipulate.  Select 
Manage  User  ASAObjects  to  see  a  list  of  users  and  a  list  of  the  configurations. 

Note:  A  configuration  in  this  case  is  a  class  (e.g.,  14-008). 

10.  From  here,  a  user  can  be  selected  and  access,  or  control  can  be  granted  to  the  user  for  any 
specific  configuration.  No  Access  will  prevent  the  user  from  accessing  the  data  from  that 
configuration.  Admin  will  grant  the  user  full  access  to  the  roster  with  no  restrictions 
(should  be  limited  to  one  or  two  users).  Author  will  allow  the  user  to  modify  the  roster  (in 
the  event  a  student  is  dropped).  Viewer  allows  the  user  to  see  the  roster,  but  the  user  is 
not  permitted  to  make  modifications. 

Note:  Each  of  the  different  levels  provides  some  degree  of  control.  Reporter  is 
largely  a  developer’s  function  and  is  not  recommended  for  use.  Admin,  Author  and 
Viewer  will  provide  the  user  with  the  appropriate  controls. 

1 1 .  Class  Roster  provides  both  an  import  function  and  an  editing  function.  Though  they  are 
presented  in  alphabetical  order,  the  likely  interaction  process  will  be  import  first,  edit 
second.  Therefore,  these  instructions  will  describe  the  functions  in  that  order. 


E-21 


12.  Import  Roster  allows  Administrators  and  Managers  to  load  a  student  roster  from  an  Excel 
file.  Select  Import  Roster  and  a  pop-up  window  will  appear.  The  window  provides  two 
options:  Select  the  Add  Files  option  or  drag  and  drop  an  Excel  file  onto  the  Drop  Files 
Here  image.  Add  Files  will  open  up  Windows  Explorer  allowing  the 
Administrator/Manager  to  navigate  to  the  location  of  the  roster  file.  If  the  file  is  already 
selected,  it  can  be  dragged  and  dropped  onto  the  Drop  Files  Here  Button  quickly.  After 
either  process,  press  Upload  and  the  roster  should  upload  into  the  database. 

Selecting  Edit  Class  Roster  presents  a  Class  selection  dropdown  and  a  simple 
filter  by  Cav  for  the  class.  With  a  Cav  selected,  all  of  the  students  in  that  Cav  will  appear. 
With  a  roster  in  the  database,  Administrators  and/or  Managers  can  edit  the  roster  as 
needed.  Editing  functions  are  currently  limited  to  adding  or  removing  a  student. 
Currently,  the  tool  does  not  allow  students  to  be  moved  from  one  Cav  to  another. 
However,  this  feature  will  be  added  in  the  future.  Note:  there  is  also  an  Add  Class  button. 
This  button  will  create  a  new  class  where  students  can  be  added  one  at  a  time.  This 
feature  is  still  in  development  and  does  not  yet  support  the  organization  of  students  into 
Cavs. 

13.  The  Instructor  Roster  menu  item  allows  Administrators  and  Managers  to  create  basic 
profiles  for  ARC  instructors.  The  Edit  Instructor  Roster  feature  functions  in  the  same 
way  as  the  Edit  Student  Roster.  Instructors  can  be  added  or  removed  from  the  database. 
Creating  these  instructor  profiles  is  essential  to  proper  data  management  as  these  profiles 
will  be  sent  to  the  ARC-FT  devices,  allowing  each  instructor  to  select  his  name  from  the 
list  to  log  in.  Note:  Instructors  are  treated  differently  than  Administrators,  Mangers,  and 
Users  in  the  Database.  This  feature  is  designed  to  enable  the  viewing  of  trending  data  by 
instructor  in  the  future. 
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Usability  Questionnaires  for  the  Tool 
ARC-FT  Usability  Scale 

Date: _  Mission/Event  Type: _ 


Directions:  Place  a  check  mark  in  the  box  that  best  represents  your  rating  of  your  experience  using 


the  ARC  Field  Tool  (ARC-FT). 

Strongly  Strongly 

Disagree  Agree 

1 .  1  think  that  1  would  like  to  use  the  ARC-FT  frequently. 

2.  1  found  the  ARC-FT  to  be  unnecessarily  complex. 

12  3  4 

12  3  4 

3.  I  thought  the  ARC-FT  was  easy  to  use. 

12  3  4 

4.  1  think  that  I  would  need  the  support  of  a  technical 

person  to  be  able  to  use  the  ARC-FT. 

12  3  4 

5.  1  found  the  various  functions  in  the  ARC-FT  to  be  well 

integrated. 

12  3  4 

6.  1  thought  there  was  too  much  inconsistency  in  the 

ARC-FT. 

12  3  4 

7.  1  would  imagine  that  most  people  would  learn  to  use 

the  ARC-FT  very  quickly. 

12  3  4 

8.  1  found  the  ARC-FT  to  be  very  cumbersome  to  use. 

12  3  4 

9.  1  felt  confident  using  the  ARC-FT. 

12  3  4 

10.  1  needed  to  learn  a  lot  of  things  before  1  could  get  going 

with  this  ARC-FT. 

12  3  4 
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Integrated  Database  Usability  Scale 


Date: 


Mission/Event  Type: 


Directions:  Place  a  check  mark  in  the  box  that  best  represents  your  rating  of  your  experience  using 
the  Integrated  Database. 


1 .  1  think  that  1  would  like  to  use  the  Integrated  Database 
frequently. 

2.  1  found  the  Integrated  Database  to  be  unnecessarily 
complex. 

3.  1  thought  the  Integrated  Database  was  easy  to  use. 

4.  I  think  that  1  would  need  the  support  of  a  technical 
person  to  be  able  to  use  the  Integrated  Database. 


Strongly  Strongly 

Disagree  Agree 


12  3  4 


5.  1  found  the  various  functions  in  the  Integrated  Database 

to  be  well  integrated.  12  3  4 


6.  1  thought  there  was  too  much  inconsistency  in  this 

Integrated  Database.  12  3  4 


7.  1  would  imagine  that  most  people  would  learn  to  use 
this  Integrated  Database  very  quickly. 


8.  1  found  the  Integrated  Database  to  be  very  cumbersome 

to  use.  12  3  4 


9.  1  felt  confident  using  the  Integrated  Database.  _ I _ I _ I _ 

12  3  4 


10.  1  needed  to  learn  a  lot  of  things  before  1  could  get  going 

with  this  Integrated  Database.  12  3  4 
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Usability  Statements  Questionnaires 

Questions  about  the  ARC-FT 

Statement 

Disagree 

Agree 

1 

Within  the  ARC-FT,  1  knew  exactly  where  to  go  to  capture  an  observation. 

2 

It  was  easy  to  tell  if  1  had  completed  an  observation. 

|  1  knew  exactly  where  to  go  within  the  tool  to  capture  an  assessment. 

|  It  was  easy  to  tell  if  1  had  completed  an  assessment. 

5 

At  times,  1  had  trouble  remembering  where  1  was  in  the  system. 

|  1  knew  when  all  of  the  outcomes  and  attributes  had  been  assessed. 

7 

Sometimes  it  was  hard  to  remember  how  to  use  the  ARC-FT. 

8 

The  ARC-FT  used  the  same  language  we  use  during  exercises. 

9 

The  terms  were  used  consistently  throughout  the  ARC-FT  tool. 

10 

1  understood  how  the  observations  and  assessments  1  made  would  be  applied  to  collect 
information  about  student  performance. 

11 

The  user  interface  supported  my  work  style  and  allowed  me  to  capture  data  in  the  way 
that  1  think  is  most  effective. 

ED 

The  ARC-FT  was  easy  to  navigate  to  different  outcomes  and  attributes. 

13 

Sometimes,  1  wanted  to  go  back  to  an  outcome,  or  attribute  to  change  my  entry,  but  it 
was  difficult  to  find  it. 

ED 

When  1  made  a  mistake,  it  was  easy  to  undo  my  error. 

ED 

1  found  that  1  made  many  errors  when  1... 

Made  camera,  voice  to  text,  or  keyboard  observations 


Used  the  roster  to  select  a  student 


Completed  an  assessment  for  a  student 


Selected  behavioral  anchors  for  attributes 


Used  the  Trending  interface 


Uploaded  data  to  the  database 


I  hardly  ever  made  an  error  when  using  the  ARC-FT  tool. 


When  I  made  an  error,  I  always  knew  it. 


It  didn't  take  me  long  to  learn  how  to... 


Make  camera,  voice  to  text,  or  keyboard  observations 


Used  the  roster  to  select  a  student 


Completed  an  assessment  for  a  student 


Selected  behavioral  anchors  for  attributes 


Used  the  Trending  interface 


Uploaded  data  to  the  database 


I  was  able  to  enter  observations  and  assessments  quickly,  even  when  I  was  using 
FT  for  the  first  time. 


The  interface  was  easy  to  read. 


The  interface  had  too  much  information  on  it. 


The  information  shown  on  the  display  was  helpful. 


I  thought  the  interface  was  organized  well. 
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Questions  about  the  Integrated  Database 

1 

2 

Statement 

Disagree 

Agree 

Within  the  Integrated  Database,  1  knew  exactly  where  to  go  to  find  student  assessments. 

It  was  easy  to  find  data  for  a  Cav  or  a  student. 

|  1  knew  exactly  where  to  go  within  the  tool  to  find  completed  assessments. 

|  It  was  easy  to  tell  if  1  had  selected  the  right  students  or  Cavs  for  trend  analysis. 

5 

At  times,  1  had  trouble  remembering  where  1  was  in  the  system. 

|  1  knew  when  all  of  the  data  were  displayed. 

7 

Sometimes  it  was  hard  to  remember  how  to  use  the  database 

8 

The  Integrated  Database  used  the  same  language  we  use  during  exercises. 

D 

The  terms  were  used  consistently  throughout  the  tool. 

1  understood  how  the  assessments  1  made  were  displayed  in  the  Integrated  Database 

ii 

The  user  interface  supported  my  work  style  and  allowed  me  to  capture  data  in  the  way 
that  1  think  is  most  effective. 

m 

The  Database  was  easy  to  navigate  to  different  Cavs  and  students. 

ES 

When  1  made  a  mistake,  it  was  easy  to  undo  my  error. 

m 

1  found  that  1  made  many  errors  when  1... 

Selected  a  Cav  or  student  from  the  filters 

Selected  Assessment  forms  for  print 

m 

1  hardly  ever  made  an  error  when  using  the  Integrated  Database. 

M 

When  1  made  an  error,  1  always  knew  it. 

m 

It  didn't  take  me  long  to  learn  how  to... 

Select  a  Cav  or  student  from  the  filters 

Select  Assessment  forms  for  print 

ES 

1  was  able  to  find  the  information  1  was  looking  for  quickly. 

m 

The  interface  was  easy  to  read. 

The  interface  had  too  much  information  on  it. 

E3 

The  information  shown  on  the  display  was  helpful. 

1  thought  the  interface  was  organized  well. 
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Interview  Protocol 


Question 

Target 

Question 

1 

Perceived 

Usability/Utility 

What  were  your  impressions  of  the  user  interface  and  the  workflow  of  the 
ARC-Field  Tool? 

2 

Perceived 

Usability/Utility 

What  aspects/components  of  the  ARC-Field  Tool  helped  you  perfonn  your  job 
more  effectively? 

3 

Perceived 

Usability/Utility 

What  aspects/components  of  the  ARC-Field  Tool  made  your  job  more  difficult? 

4 

Feature 

Requests 

What  observation,  assessment,  and  debriefing  tasks  do  you  need  help  doing? 
Could  some  change  to  the  ARC-Field  Tool  or  Integrated  Database  help  you  do 
them? 

5 

Use 

Would  you  use  the  ARC  Perfonnance  Assessment  Toolset?  Do  the  ARC-FT  and 
Integrated  Database  meet  the  needs  of  the  ARC? 

6 

Feature 

Requests 

How  can  we  improve  the  ARC-Field  Tool?  How  can  we  improve  the  Integrated 
Database? 

H-l 


APPENDIX  I 


Item-Level  Descriptive  Statistics  for  Tool  Evaluation 


Responses  to  Standard  Usability  Questions  for  ARC-FT  Supported  Evaluation 


Statement 

N 

Mean 

Median 

Mode 

1 .  I  think  that  I  would  like  to  use  these  display  concepts 

frequently. 

7 

3.71 

4 

4 

2.  I  found  the  system  to  be  unnecessarily  complex.* 

7 

1.29 

1 

1 

3.  I  thought  the  system  was  easy  to  use. 

7 

3.71 

4 

4 

4.  I  think  that  I  would  need  the  support  of  a  technical 

person  to  be  able  to  use  this  system.* 

7 

1.29 

1 

1 

5.  I  found  the  various  functions  in  this  system  to  be  well 

integrated. 

7 

3.57 

4 

4 

6.  I  thought  there  was  too  much  inconsistency  in  this 

system.* 

7 

1.57 

1 

1 

7.  I  would  imagine  that  most  people  would  leam  to  use 

this  system  very  quickly. 

7 

3.71 

4 

4 

8.  I  found  the  system  to  be  very  cumbersome  to  use.* 

6 

1.33 

1 

1 

9.  I  felt  confident  using  the  system. 

7 

3.86 

4 

4 

10.  I  needed  to  leam  a  lot  of  things  before  I  could  get 
going  with  this  system.* 

7 

1.57 

1 

1 

*  Reverse  coded  items 


Responses  to  Standard  Usability  Questions  for  ARC-FT  Unsupported  Evaluation 


Statement 

N 

Mean 

Median 

Mode 

1. 

I  think  that  I  would  like  to  use  this  ARC-FT 

frequently. 

5 

3.40 

3 

3 

2. 

I  found  the  ARC-FT  to  be  unnecessarily 

complex.* 

5 

2.00 

2 

2 

3. 

I  thought  the  ARC-FT  was  easy  to  use. 

5 

3.20 

3 

4 

4. 

I  think  that  I  would  need  the  support  of  a 
technical  person  to  be  able  to  use  this  ARC-FT.* 

5 

1.60 

1 

1 

5. 

I  found  the  various  functions  in  this  ARC-FT 
to  be  well  integrated. 

5 

3.40 

3 

3 

6. 

I  thought  there  was  too  much  inconsistency  in 
this  ARC-FT.* 

5 

1.60 

2 

2 

7. 

I  would  imagine  that  most  people  would  leam 
to  use  this  ARC-FT  very  quickly. 

5 

3.60 

4 

4 

8. 

I  found  the  ARC-FT  to  be  very  cumbersome 

5 

1.80 

2 

2 
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to  use.* 

9.  I  felt  confident  using  the  ARC-FT. 

5 

3.60 

4 

4 

10.  I  needed  to  learn  a  lot  of  things  before  I  could 

get  going  with  this  ARC-FT.* 

5 

1.60 

2 

2 

*  Reverse  coded  items 

Responses  to  Standard  Usability  Questions  for  ARC -DB  Unsupported  Evaluation 

Statement 

N 

Mean  Median  Mode+ 

1 .  I  think  that  I  would  like  to  use  this  Integrated  Database 
frequently. 

3 

3.00 

3 

2.  I  found  the  Integrated  Database  to  be 

unnecessarily  complex.* 

3 

2.00 

2 

3.  I  thought  the  Integrated  Database  was  easy  to 

use. 

3 

3.00 

3 

4.  I  think  that  I  would  need  the  support  of  a 

technical  person  to  be  able  to  use  this  Integrated 
Database.* 

3 

2.00 

2 

5.  I  found  the  various  functions  in  this 

Integrated  Database  to  be  well  integrated. 

2 

3.00 

3 

' 

6.  I  thought  there  was  too  much  inconsistency 

in  this  Integrated  Database.* 

2 

2.00 

2 

' 

7.  I  would  imagine  that  most  people  would 

learn  to  use  this  Integrated  Database  very  quickly. 

3 

3.00 

3 

— 

8.  I  found  the  Integrated  Database  to  be  very 

cumbersome  to  use.* 

3 

2.00 

2 

— 

9.  I  felt  confident  using  the  Integrated 

Database. 

3 

2.67 

3 

' 

10.  I  needed  to  leam  a  lot  of  things  before  I 

could  get  going  with  this  Integrated  Database.* 

3 

2.33 

2 

' 

+  No  mode  was  calculated  due  to  small  n. 

*  Reverse  coded  items 

Responses  to  Usability  Statemen  ts  for  ARC-FT  Supported  Field  Evaluation 

Statement 

n 

%  agree 

1  I  knew  exactly  where  to  go  to  capture  an  observation. 

7 

100% 

2  It  was  easy  to  tell  if  I  had  completed  an  observation. 

7 

57% 

3  I  knew  exactly  where  to  go  to  capture  an  assessment. 

7 

100% 

4  It  was  easy  to  tell  if  I  had  completed  an  assessment. 

7 

71% 
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5 

At  times,  I  had  trouble  remembering  where  I  was  in  the 
system.* 

7 

29% 

6 

I  knew  when  all  of  the  outcomes  and  attributes  had  been 
assessed. 

7 

86% 

7 

Sometimes,  it  was  hard  to  remember  how  to  use  the 
tool.* 

7 

43% 

8 

The  ARC-FT  tool  used  the  same  language  we  use  during 
exercises. 

5 

100% 

9 

The  tenns  were  used  consistently  throughout  the  ARC- 
FT  tool. 

7 

100% 

10 

I  understood  how  the  observations  and  assessments 
would  be  applied  to  collect  information  about  student 
perfonnance. 

7 

86% 

11 

The  user  interface  supported  my  work  style  and  allowed 
me  to  capture  data  in  the  way  that  I  think  is  most 
effective. 

7 

100% 

12 

It  was  easy  to  navigate  to  different  outcomes  and 
attributes. 

7 

100% 

13 

Sometimes,  I  wanted  to  go  back  to  an  outcome,  or 
attribute  to  change  my  entry,  but  it  was  difficult  to  find 
it.* 

7 

29% 

14 

When  I  made  a  mistake,  it  was  easy  to  undo  my  error. 

7 

57% 

15 

I  found  that  I  made  many  errors  when  I. . .  * 

- 

- 

Made  camera,  voice  to  text,  or  keyboard 
observations 

6 

50% 

Used  the  roster  to  select  a  student 

6 

33% 

Completed  an  assessment  for  a  student 

6 

50% 

Selected  behavioral  anchors  for  attributes 

6 

17% 

Used  the  Trending  interface 

6 

0% 

Uploaded  data  to  the  database 

4 

25% 

16 

I  hardly  ever  made  an  error  when  using  the  ARC-FT 
tool. 

7 

43% 

17 

When  I  made  an  error,  I  always  knew  it. 

7 

29% 

18 

It  didn’t  take  me  long  to  leam  how  to 

- 

- 

Made  camera,  voice  to  text,  or  keyboard 
observations 

7 

100% 

Used  the  roster  to  select  a  student 

7 

100% 

Completed  an  assessment  for  a  student 

7 

100% 

Selected  behavioral  anchors  for  attributes 

7 

71% 

Used  the  Trending  interface 

7 

57% 

Uploaded  data  to  the  database 

5 

60% 

19 

I  was  able  to  enter  observations  and  assessments  quickly, 
even  when  I  was  using  the  ARC-FT  tool  for  the  first 
time. 

7 

86% 

20 

The  interface  was  easy  to  read. 

7 

100% 

21 

The  interface  had  too  much  information  on  it.* 

7 

14% 
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22  The  information  shown  on  the  display  was  helpful. 

7 

100% 

23  I  thought  the  interface  was  organized  well. 

7 

100% 

*  Reverse  coded  items 

Responses  to  Usability  Statements  for  ARC-FT  Unsupported  Evaluation 


Statement 

n 

% 

agree 

1 

Within  the  ARC-FT,  I  knew  exactly  where  to  go  to  capture  an 
observation. 

5 

60% 

2 

It  was  easy  to  tell  if  I  had  completed  an  observation. 

4 

75% 

3 

I  knew  exactly  where  to  go  within  the  tool  to  capture  an 
assessment. 

4 

100% 

4 

It  was  easy  to  tell  if  I  had  completed  an  assessment. 

4 

75% 

5 

At  times,  I  had  trouble  remembering  where  I  was  in  the 
system.* 

4 

50% 

6 

I  knew  when  all  of  the  outcomes  and  attributes  had  been 
assessed. 

4 

100% 

7 

Sometimes  it  was  hard  to  remember  how  to  use  the  ARC-FT.* 

4 

25% 

8 

The  ARC-FT  used  the  same  language  we  use  during  exercises. 

4 

100% 

9 

The  terms  were  used  consistently  throughout  the  ARC-FT  tool. 

4 

100% 

10 

I  understood  how  the  observations  and  assessments  I  made 
would  be  applied  to  collect  information  about  student 
perfonnance. 

4 

100% 

11 

The  user  interface  supported  my  work  style  and  allowed  me  to 
capture  data  in  the  way  that  I  think  is  most  effective. 

4 

100% 

12 

The  ARC-FT  was  easy  to  navigate  to  different  outcomes  and 
attributes. 

4 

100% 

13 

Sometimes,  I  wanted  to  go  back  to  an  outcome,  or  attribute  to 
change  my  entry,  but  it  was  difficult  to  find  it.* 

4 

0% 

14 

When  I  made  a  mistake,  it  was  easy  to  undo  my  error. 

4 

75% 

15 

I  found  that  I  made  many  errors  when  I. . .  * 

- 

- 

Made  camera,  voice  to  text,  or  keyboard  observations 

4 

50% 

Used  the  roster  to  select  a  student 

4 

0% 

Completed  an  assessment  for  a  student 

4 

25% 

Selected  behavioral  anchors  for  attributes 

4 

0% 

Used  the  Trending  interface 

4 

25% 

Uploaded  data  to  the  database 

3 

33% 

16 

I  hardly  ever  made  an  error  when  using  the  ARC-FT  tool. 

4 

50% 

17 

When  I  made  an  error,  I  always  knew  it. 

4 

100% 

18 

It  didn’t  take  me  long  to  leam  how  to 

- 

- 

Make  camera,  voice  to  text,  or  keyboard  observations 

4 

50% 

Used  the  roster  to  select  a  student 

4 

100% 

Completed  an  assessment  for  a  student 

4 

100% 

Selected  behavioral  anchors  for  attributes 

4 

100% 

Used  the  Trending  interface 

3 

100% 

Uploaded  data  to  the  database 

3 

67% 
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19 

I  was  able  to  enter  observations  and  assessments  quickly,  even 
when  I  was  using  the  ARC -FT  for  the  first  time. 

4 

75% 

20 

The  interface  was  easy  to  read. 

4 

100% 

21 

The  interface  had  too  much  information  on  it.* 

3 

67% 

22 

The  information  shown  on  the  display  was  helpful. 

4 

100% 

23 

I  thought  the  interface  was  organized  well. 

4 

75% 

*  Reverse  coded  items 

Quantitative  Results  of  Responses  to  Usability  Statements  for  ARC  Database  Unsupported 
Evaluation 


Statement  n  % 


agree 


1 

Within  the  Integrated  Database  ,  I  knew  exactly  where  to  go  to 
find  student  assessments 

3 

33% 

2 

It  was  easy  to  data  for  a  Cav  or  a  student. 

3 

33% 

3 

I  knew  exactly  where  to  go  within  the  tool  to  find  completed 
assessments. 

3 

67% 

4 

It  was  easy  to  tell  if  I  had  selected  the  right  students  or  Cavs  for 
trend  analysis. 

2 

100% 

5 

At  times,  I  had  trouble  remembering  where  I  was  in  the 
system.* 

2 

0% 

6 

I  knew  when  all  of  the  data  were  displaying.* 

2 

50% 

7 

Sometimes  it  was  hard  to  remember  how  to  use  the  database.* 

2 

0% 

8 

The  Integrated  Database  used  the  same  language  we  use  during 
exercises. 

2 

100% 

9 

The  terms  were  used  consistently  throughout  the  tool. 

2 

100% 

10 

I  understood  how  the  assessments  I  made  were  displayed  in  the 
Integrated  Database 

2 

100% 

11 

The  user  interface  supported  my  work  style  and  allowed  me  to 
capture  data  in  the  way  that  I  think  is  most  effective. 

2 

100% 

12 

The  Database  was  easy  to  navigate  to  different  Cavs  and 
students. 

2 

100% 

13 

When  I  made  a  mistake,  it  was  easy  to  undo  my  error. 

2 

50% 

14 

I  found  that  I  made  many  errors  when  I. . .  * 

Selected  a  Cav  or  student  from  the  filters 

2 

0% 

Selected  Assessment  forms  for  print 

2 

0% 

15 

I  hardly  ever  made  an  error  when  using  the  Integrated 

Database. 

2 

50% 

16 

When  I  made  an  error,  I  always  knew  it. 

2 

100% 

17 

It  didn’t  take  me  long  to  leam  how  to. . . 

Selected  a  Cav  or  student  from  the  filters 

2 

100% 

Selected  Assessment  forms  for  print 

2 

50% 

18 

I  was  able  to  find  the  information  I  was  looking  for  quickly. 

2 

50% 

19 

The  interface  was  easy  to  read. 

2 

100% 

20 

The  interface  had  too  much  information  on  it.* 

2 

50% 
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21  The  information  shown  on  the  display  was  helpful. 

2 

100% 

22  I  thought  the  interface  was  organized  well. 

2 

*  Reverse  coded  items 
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